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CHAPTER  1 


INTRODUCTION 


1.1.  Statement  of  Problem 

This  paper  considers  the  problem  of  finding  optimal  policies  for 
several  deteriorating  systems.  The  systems  discussed  here  include  deter- 
iorating machinery  whose  deterioration  can  be  observed  by  a decision  maker 
or  any  system  that  involves  periodic  repair.  Because  of  the  applicability 
of  these  systems,  a number  of  authors  have  contributed  to  the  development 
of  the  optimization  theory  for  the  discrete  time  maintenance  models.  Almost 
all  of  them  are  replacement  models  rather  than  repair  models  in  the  sense 
that  an  unlimited  supply  of  new  spares  is  available  and  the  amount  of  time 
needed  for  the  replacement  or  repair  is  negligibly  small.  In  practice  we 
often  encounter  the  cases  where  there  are  a limited  number  of  spare  units 
and  where  the  repair  time  of  a machine  is  no  longer  negligible  as  compared 
with  the  length  of  its  operating  time.  Repair  time  of  a machine  may  even 
depend  on  its  degree  of  deterioration.  Several  examples  of  such  processes 
can  be  easily  introduced.  Aircraft  repair  problem  of  an  airline  company, 
automobile  repair  problem  of  a taxicab  company  and  factory  machine  repair 
problem  are  some  of  them.  In  particular,  the  problem  of  determining  the 
date  and  quantity  of  suits  one  should  bring  to  a laundry  for  dry  cleaning 
can  be  applicable  if  cleanliness  of  a suit  can  be  somehow  measured. 


Since  repair  rather  than  replacement  is  often  applicable,  we  have  developed 
some  mathematical  models  which  incorporate  maintenance  with  repair. 

The  discrete  time  Markov  maintenance  problem  with  repair  can  be 
stated  in  the  following  way.  A machine  is  assumed  to  be  operating  over 
time  with  its  condition  deteriorating  as  time  goes  on.  It  is  observed  at 
discrete  time  intervals  and  is  classified  as  being  in  one  of  1+1  states 
{0, 1,2, . . . , 1} . State  0 represents  a machine  in  perfect  working  order, 
and  f 1,2, , . . , 1-1}  represents  intermediate  states  of  deterioration  lead- 
ing to  failure  at  state  I.  An  operating  machine  can  be  sent  to  a repair 
shop  at  the  beginning  of  each  period  just  after  the  observation  of  its 
state,  whereas  a failed  machine  must  be  repaired.  When  a machine  is 
being  repaired,  the  number  of  time  periods  that  it  is  unavailable  is 
usually  assumed  to  have  a geometric  distribution,  A repaired  machine 
becomes  available  in  its  best  condition.  If  the  repair  decision  is  not 
chosen  at  the  beginning  of  a period,  the  machine  in  the  i-th  operating 
condition  keeps  operating  and  at  the  beginning  of  the  next  period  it 
moves  to  the  i-th  operating  condition  with  stationary  transition  probability 
p.  ,.  An  operating  cost  is  charged  while  a machine  is  operating,  and 
material  and  labor  costs  are  charged  when  it  is  being  repaired.  These 
costs  usually  depend  on  the  state  of  the  machine,  The  objective  is  to 
find  a repair  policy  which  minimizes  the  total  expected  a-discounted 
cost,  or  the  long-run  expected  average  cost  per  unit  period. 
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There  is  a simple  repair  rule  called  a control  limit  policy: 

Repair  the  machine  if  and  only  if  its  observed  state  is  greater  than 
or  equal  to  some  control  limit  i . Control  limit  rules  are  intui- 
tively appealing,  but  are  not  optimal  in  all  cases.,  Certain  restrictions 
must  be  placed  on  both  the  cost  structure  and  the  transition  probabilities 
governing  deterioration.  Chapter  2 investigates  sufficient  conditions 
under  which  an  optimal  policy  is  assured  to  be  of  a control  limit  form. 

The  aforementioned  model  had  only  one  machine  in  the  system. 

In  Chapter  3 models  with  spare  machines  in  the  system  are  studied.  In 
these  models  when  a decision  maker  decides  to  repair  an  operating 
machine,  it  is  sent  to  a repair  shop  immediately  and  replaced  by  a spare 
machine,  if  any  is  available.  Replacement  is  assumed  to  take  one  period 
of  time  just  for  mathematical  simplicity.  Machines  in  the  repair  shop 
are  independently  repaired  with  geometric  repair  time  distributions. 

The  system  fails  when  all  the  machines  are  in  the  repair  shop  and  in- 
operative, A penalty  cost  is  then  assessed.  Special  emphasis  is  being 
placed  on  finding  the  sufficient  conditions  which  result  in  the 
optimality  of  control  limit  policies  of  some  kind. 

Chapter  4 presents  some  recursive  algorithms  for  calculating 
the  optimal  cost  and  its  corresponding  policy.  Lastly,  an  explicit 
calculation  of  an  optimal  control  limit  policy  is  obtained  for  systems 
with  few  machines  and  with  simple  cost  structures  and  transition  prob- 
abilities. 
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1.2.  Brief  Survey  of  Discrete  Time  Maintenance  Models 


1 


l 

i 

! 


I 


In  this  section  we  give  a brief  review  of  previous  studies  that 
relate  to  our  work.  For  almost  two  decades  there  has  been  a great  deal 
of  interest  in  the  study  of  maintenance  problems  because  of  their  appli- 
cability in  the  practical  world.  Maintenance  models  are  usually  divided 
into  discrete  time  models  and  continuous  time  models.  As  our  study  is 
in  the  category  of  discrete  time  models,  continuous  time  maintenance 
models  will  not  be  taken  into  consideration  here. 

Discrete  time  maintenance  models  are  the  models  which  use  the 
information  regarding  the  state  of  the  system  such  as  the  degree  of 
deterioration  of  the  unit  in  order  to  choose  the  best  maintenance  action 
at  discrete  time  points.  In  1963  Derman  [7]  introduced  the  basic  model 
of  this  type.  In  his  model,  there  is  a unit  in  a system  which  is  in- 
spected periodically.  After  each  inspection  it  is  classified  in  one  of 
1+1  states,  0,1, ...,I,  showing  the  degree  of  deterioration.  There  are 
two  actions  available  at  each  period.  If  replacement  action  is  taken, 
the  unit  is  replaced  by  a new  unit  and  the  new  unit  begins  to  operate 
in  its  best  condition  at  the  beginning  of  the  next  period.  If  the 

action,  not  to  replace,  is  chosen,  the  unit  keeps  operating.  Then  P, 

* I 

is  the  probability  that  the  state  of  the  unit  moves  from  i to  j in 
one  unit  of  time.  The  cost  to  replace  an  operative  unit  and  the  cost 
of  replacing  an  inoperative  unit,  i.e.,  a unit  in  state  I,  are 
introduced.  The  latter  is  assumed  to  be  greater  than  the  former. 

Derman  showed  the  existence  of  a control  limit  policy  optimizing  the 


total  a-discounted  cost  and/or  the  long-run  average  cost  under  reason- 
able conditions  on  the  transition  probabilities. 

There  are  several  directions  for  extending  Derman's  basic  model. 
Kolesar  [l4]  extended  the  basic  model  by  introducing  state  dependent 
operating  cost  without  changing  the  basic  conclusion  of  the  model. 
Kalymon  [11]  further  generalized  the  cost  structure  by  allowing  replace- 
ment costs  to  be  stochastic,  retaining  the  optimality  of  the  control 
limit  policies. 

Another  expansion  of  Derman's  model  with  regard  to  the  cost 
structure  is  the  introduction  of  an  inspection  cost  by  Klein  [13]. 

He  assumed  that  the  condition  of  the  system  is  known  only  when  it  is 
inspected,  which  is  costly.  At  each  period  the  decision  maker  has  two 
basic  alternatives:  to  replace  the  system  or  to  keep  it.  When  the 
latter  is  selected,  he  must  further  decide  the  extent  of  repairs  to  be 
made  and  when  to  make  the  next  inspection.  He  formulated  the  model 
as  a LP  problem.  Derman  [8]  later  showed  that  the  optimal  solution 
is  a stationary  policy.  Eppen  [10],  Taylor  [20]  and  Rosenfield  [17] 
developed  their  models  along  this  line  with  emphasis  on  trying  to  find 
simple  types  of  policies  which  optimize  their  models. 

Derman  and  Lieberman  [9]  considered  a joint  replacement  and 
stocking  problem,  which  was  generalized  by  Ross  [18]  to  allow  for  a 
deterioration  of  a unit  from  time  to  time.  Ross  also  extended  the 
state  space  to  be  continuous  or  denumerable.  Another  expansion  of  the 
basic  model  with  respect  to  the  state  space  was  made  by  Wagner  [22], 


who  considered  two  modes  of  failure. 


As  was  mentioned  earlier,  the  distinction  between  repair  and 


e 


replacement  was  first  considered  by  Klein  [13],  In  his  model  an 
immediate  transition  to  the  best  state  takes  place  when  a replacement 
is  made,  while  an  immediate  transition  to  one  of  the  better  states 
takes  place  (depending  on  the  extent  of  the  repair)  when  a repair  is 
made.  In  1973  Kao  [12]  introduced  a semi -Markovian  approach  to  the 
basic  model.  He  allowed  the  time  spent  in  each  state  before  a trans- 
ition takes  place  to  be  a random  variable  depending  on  the  transition. 
Therefore  in  his  model  repair  is  no  longer  instantaneous  but  takes 
some  random  time.  Kao  found  sufficient  conditions  for  the  optimality 
of  control  limit  policies  over  the  class  of  stationary  non-randomized 
policies.  Among  them  are  state  independent  expected  repair  time  and 
repair  costs,  which  could  be  considered  as  restrictive  conditions. 

Derman  [6]  himself  considered  his  basic  model  using  a 
different  objective  function.  He  maximized  the  expected  length  of 
time  between  replacements  subject  to  the  conditions  that  the  prob- 
abilities of  replacement  through  certain  undesirable  states  are  bounded. 
Kolesar  [15]  showed  the  control  limit  optimality  for  this  case. 


1.3.  Mathematical  Background 

Before  proceeding  further  it  is  well  to  describe  some  basic 
terms  and  fundamental  theorems  frequently  used  in  our  report.  A Markov 
decision  process  is  discussed  first  since  this  theory  is  the  primary 
approach  used  in  each  model  formulation. 
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Suppose  a system  is  observed  at  points  of  time  t = 0,1,... 
and  classified  into  one  of  a number  of  possible  states  labeled 
0,1, ...,I.  After  each  observation  of  the  state  of  the  system,  an 
action  must  be  selected.  The  set  of  all  possible  actions  is  denoted 
by  A,  assumed  finite.  If  action  a 6 A is  chosen  when  the  state 

of  the  system  is  i at  time  t,  a cost  C(i,a)  is  incurred,  and  the 

next  state  is  determined  according  to 

£[Xt+l  = 0|Xo,ao,X1,a1,...,  Xt  = i,  at  = a}  = p..(a)  , 

where  X is  the  state  of  the  system  at  time  t,  and  a is  the 

action  chosen  at  t.  A rule  or  policy  is  a prescription  for  taking 

actions  at  each  time.  It  may  be  randomized  or  it  may  depend  on  the 

past  history  of  the  process.  A policy  is  called  stationary  if  it 

is  nonrandomized,  and  the  action  it  chooses  at  t depends  only  on 

the  state  of  the  system  at  t.  Let  R = (f, f, ...)  be  a stationary 

policy  whose  action  is  f(i)  when  the  state  of  the  system  is  i.  Then 

it  is  known  that  a process  modified  by  f is  a Markov  chain  with  the 

transition  probability  matix  (p..(f(i))}. 

— J 

Two  criteria  are  of  interest  as  a suitable  cost  to  be 

minimized.  The  first  criterion  is  the  total  expected  a-discounted 

cost.  Let  a 6 (0,1).  The  total  a-discounted  cost  when  a policy 

R is  employed  is  represented  by 

00 

VB,q(1)  ■ V. T-  “ C<Vat>  lX0  “ ‘l  > 0 < 1 < X" 
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where  ER  means  the  conditional  expectation  given  that  the  policy 
R is  employed.  Let 


t'V  --i 


Vc(i)  ■ ‘f  ’ 


1 0,1, 


The  following  theorems  hold. 


Theorem  1.1  (see  Ross  [19])=  is  the  unique  solution  to 


V (i)  = min{C(i,a)  + a E p .(a)  V (j) }, 
U a j=0  a 


i =0,1,...,!. 


Furthermore,  if  R*  = (f  ,f  , ...)  is  the  stationary  policy  satisfying 


c(i,fa(i))  ♦ a ^ vo(J) 


= minf C(i, a)  + a E p.  (a)  V (j)' 
a j=0  ^ a 


for  each  i then 


= V1’  ’ 


i - 0,1,..,,!. 


The  first  part  of  the  above  theorem  gives  a functional 

equation  satisfied  by  the  optimal  cost  function  V , while  the 

a 

last  part  guarantees  the  existence  of  an  optimal  a-discounted 


Policy,  Furthermore  it  may  he  taken  to  De  stationery  as  has  been 
shown  in  Derman  [5],  Wagner  [21]  and  Blackwell  [3], 


Theorem  1,2  (Bellman  [2]).  Let  V (i,n)  he  the  n period  optimal 

Ul 

expected  a-discounted  cost  starting  from  state  i Then 


V (i) 
a ' 


lim 
n -» oo 


V 

OL 


(i,n)  , 


i - 0,1, 


I 


Theorem  1,2  shows  the  validity  of  formulating  the  infinite 
horizon  problem  as  the  n period  problem,  and  then  letting  n -»«. 

The  long-run  expected  average  cost  when  a policy  R is 
employed  is  expressed  as 


VD  - 

The  policy 


1 T“1 

lim  E [ - Z C(X  a )|X  = i],  i = 0,1, 

T-.00  ^ 1 t=0  u 


■K- 

R is  said  to  be  average  cost  optimal  if 


,1. 


Vfi+(i)  = min  VR(i),  i 0,1,  . 

R 

The  existence  of  such  a policy  is  not  guaranteed  in  general, 
but  the  following  theorem  gives  a sufficient  condition  for  the 
optimality  of  a stationary  policy  as  well  as  for  the  existence  of 
an  optimal  policy. 

Let  7r  be  the  class  of  all  policies,  and  let  n'  be  the 
class  of  all  stationary  policies. 
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Assumption  1.  For  all  R € v’ , the  modified  Markov  chain  by  R 
is  irreducible. 

Theorem  1.3  (see  Ross  [19]).  It  Assumption  1 holds,  then  there 
exists  R € it'  such  that 

VR*(i)  = min  VR(i)  = min  V (i)  = lim  (l-a)  V (0), 

R£tt  RStt'  a -» 1 

Theorem  1.3  also  shows  that  the  optimal  average  cost  is 
independent  of  the  starting  state  of  the  process  and  is  related  to 
the  total  a-discounted  cost.  Therefore  the  results  obtained  for  the 
total  a-discounted  cost  can  be  often  applied  to  the  expected  average 
cost  case  through  this  theorem.  The  irreducibility  assumption, 
however,  seems  too  strong  for  our  models.  Fortunately  the  next 
assumption  suffices. 

Assumption  2.  For  all  R € tt',  there  exists  a special  state,  say 

i , which  is  accessible  from  every  other  state, 
s 

Theorem  1.4  (Ross  [19]).  Theorem  1.3  still  holds  after  replacing 
Assumption  1 with  Assumption  2. 

In  reliability  theory  where  the  state  of  a system  is 
represented  by  being  good  or  bad,  increasing  failure  rate  (IFR) 
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distributions  play  an  important  role  (see  Barlow  [1]).  An  item  with 
an  IFR  failure  distribution  has  the  property  of  aging  and  is  more  likely 
to  fail  the  older  it  gets.  Derman  [7]  developed  the  notion  of  an  IFR 
distribution  for  a Markov  cnain  with  a finite  number  of  states.  Suppose 
there  are  1+1  states  indexed  to  show  the  degree  of  deterioration.  Then 
a Markov  chain  is  said  to  be  IFR  if  the  higher  the  state  the  greater 
the  chance  of  further  deterioration.  More  precisely,  a Markov  chain 
with  the  transition  matrix  fp.  .)  is  said  to  be  IFR  if  P.(°)  is 

i«r  1 

stochastically  smaller  than  (• ) for  i = 0,1,..., 1-1,  and  can 

be  written  as 


Pi(')  = Pi+i(-),  i = 0,1,. ..,1-1, 

where 

*4.00=  I p,,,  i = 0,i,...,i. 

j<k  1J 

Note  that  suppose  F(t)  and  G(t)  are  distribution  functions, 

F(")  ^ G(')  if  and  only  if  F(t)  > G(t)  for  any  t.  Since  we  are 
concerned  with  maintenance  problems,  aging  of  a machine  is  a reasonable 
assumption,  and  hence,  IFR  distributions  will  be  assumed  in  each  of 
our  models.  Lastly  a useful  lemma  is  stated. 

Lemma  1.5  (Derman  [7]). 

Pi(-)  c Pin(')>  i « 0,1,. ..,1-1, 

if  and  only  if  for  every  nondecreasing  function  h(j),  j = 0, ...,I, 

H(i)  = Z1  p h(j)  is  nondecreasing  in  i for  0 < i < I. 

-J  “V/  1-1 
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CHAPTER  2 


MAINTENANCE  MODELS  WITHOUT  SPARE  UNITS 

In  this  chapter  we  present  several  machine  repair  models  where 
there  is  only  one  machine  in  the  system.  The  first  section  treats  the 
special  case  where  Loth  the  repair  time  and  labor  costs  are  independent 
of  the  state  of  the  machine,  while  the  last  section  treats  the  general 
case  where  they  depend  on  the  state  of  the  machine.  In  each  section 
obtaining  sufficient  conditions  for  the  optimality  of  a control  limit 
policy  is  of  special  interest.  In  the  second  section  the  results  in  the 
first  section  are  shown  to  be  applicable  to  the  case  where  the  repair 
time  distribution  is  no  longer  a geometric  distribution. 

2.1.  Special  Case 

Consider  a discrete  time  machine  repair  problem  which  is 
schematically  shown  in  Fig.  2.1.  There  is  a machine  in  the  system  with 

its  condition  deteriorating  as  time 
goes  on.  It  is  observed  at  the 
beginning  of  each  period,  and  is 
classified  as  being  in  one  of  1+1 
(I  > 1)  states.  State  0 represents 

Figure  2.1:  A Machine  Repair 

System  Without 
Spare  Units. 


i 

I 


a machine  in  its  best  condition,  while  { 1,2, . . . , 1-1)  denote  inter- 
mediate states  of  deterioration  leading  to  failure  at  state  I.  Also 
a state  1+1  is  introduced,  denoting  that  a machine  is  under  repair. 

When  a machine  is  operating,  two  choices  are  available  just  after 
each  observation  of  its  state  by  a decision  maker:  to  let  it  keep 

operating,  or  to  send  it  to  the  repair  shop.  If  the  former  decision 
is  selected,  the  state  of  the  machine  evolves  from  i to  j in  one 
unit  of  time  according  to  the  transition  probability  p_  >0.  We 
-j.  pik  = 1 for  any  i throughout  this  paper.  If  the 
latter  decision  is  chosen,  the  machine  is  immediately  sent  to  the 
repair  shop,  and  stays  there  until  it  is  completely  repaired.  A failed 
machine  must  be  repaired.  The  number  of  time  periods  taken  to  complete 
the  repair  work  is  assumed  to  have  a geometric  distribution  independent 
of  the  state  of  the  machine  to  be  repaired.  More  precisely,  let 
be  a repair  time  of  a machine  in  state  i.  Then 


assume 


P(T1  - j)  - q(l-q)J"\ 


0 — 2 y • • O 


f 4 

L 


i 


i j i 


Thus,  a machine  in  the  repair  shop  is  completely  repaired  by  the  end 
of  that  period  with  probability  q.  Assume  0 < q < 1.  A repaired 
machine  becomes  available  in  its  best  condition. 

The  costs  involved  in  this  system  are: 

A(i):  operating  cost  for  a machine  in  state  i (0  < i < I). 

C(i):  material  cost  for  repairing  a machine  in  state  i (0  < i < I). 

B:  labor  cost  for  repairing  a machine  per  unit  period. 


I 
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Note  that  the  statement  that  a failed  machine  must  be  repaired  is 
satisfied  if  we  set  A(l)  to  be  infinity  or  a very  large  number. 
Material  cost  is  assumed  to  be  charged  a.t  the  start  of  the  repair  work. 
The  objective  is  to  find  a repair  policy  which  minimizes  the  total 
a-discounted  cost. 

This  system  is  a Markov  decision  process  and  hence,  can  be 

formulated  as  a dynamic  programming  problem.  Let  V (i,n)  be  the 

a 

expected  n period  a-discounted  minimum  cost  starting  from  state  i at  the 
beginning.  Then  V (i;n)  satisfies  a set  of  recursive  equations: 

/■** 

V (i;l)  = minfA(i),  C(i)+B),  0 < i < 1 

V (1+1:1)  ^ B 
a 5 ' 

I 

i 

/ For  n > 1, 

! i 

Vr><'i’n+i)  “ “^M1)  + a S P.  V (j,n),  C(i)  + B (2.1) 

j=o  10  a 

| 

+ a(qVa(0,n)  + (1-q)  Va(l+l;n) ) ) , 

° < i < 1 

V (l+ljn+1)  = B + a(qV  (0,n)  + (1-q)  V Jl+l,n))  . 

W UL  UL 

Let  V^(i)  be  the  total  expected  a-discounted  cost  starting 
from  state  i.  Then  by  Theorem  1.1  the  existence  of  a stationary  policy 


! 


minimizing  the  total  a-discounted  cost  is  guaranteed.  Now  a special 
subclass  of  a set  of  policies,  called  a control  limit  policy,  is  of 
interest,  which  is  defined  as  below; 

Definition.  A control  limit  policy  is  a nonrandomized  policy  where 
there  is  an  i for  each  period  n,  say  i^,  such  that  for  all  i with 
i < i , the  decision  at  period  n is  to  keep  a machine  in  operation, 
and  for  all  i with  i > i , the  decision  at  period  n is  to  repair  it. 
It  is  said  to  be  stationary  if  the  i ' s,  called  the  control  limits, 
are  constant  in  n. 

A control  limit  policy  has  a very  simple  structure  and  is 
intuitively  appealing  as  a rule  that  an  optimal  policy  must  satisfy. 
Furthermore,  if  an  optimal  policy  is  known  to  possess  this  property, 
the  calculation  of  obtaining  the  optimal  policy  can  be  significantly 
simplified.  But  in  order  to  assure  that  a control  limit  policy  is 
optimal,  certain  restrictions  must  be  placed  on  both  the  cost  structure 
and  the  transition  probabilities  governing  deterioration.  Investigated 
here  are  sufficient  conditions  under  which  an  optimal  policy  is  assured 
to  be  of  a control  limit  form.,  First  consider  a finite  horizon  problem. 
For  any  natural  number  N,  we  have  the  following  results; 

Theorem  2.1.  Assume  the  following  conditions  hold; 

1.  C ( i ) is  nondecreasing  in  i for  0 < i < I. 

2.  A(i)  - C(i)  is  nondecreasing  in  i for  0 < i < I. 

5.  P^OcP  (•)  for  0 < i < 1-1. 
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Then  there  exists  a control  limit  policy  which  minimizes  the  N- stage 
a- discounted  cost  of  the  model, 


Proof:  We  first  show  that  V (i;n)  is  nondecreasing  in  i for 

0 < 1 I for  each  1 _ n < N,  Proof  is  by  mathematical  induction. 
Look  at  (2,1)  and  notice  that  V^(i.l)  is  trivially  no.udecreasing  in 

1 oince  both  A(i)  and  C(i)  are  nondecreasing  in  : from  1 and 

Suppose  V^(i;n)  is  nondecreasing  in  i (0  < i < I)  for  1 ^ n < N-l 
Then  from  J>  and  by  the  inductive  assumption,  Q p.  V^(j;n)  is  non- 
decreasing in  i using  Lemma  1,5,  and  hence  A ( i ) - cr  > 1 „ p.  V ( -)s  n) 

.1=0  ri(]  a * 

is  nondecreasing  in  i (0  < i < I),  Also  from  condition  1, 

C'i)  + B + a(qV  (0|n)  + (1-q)  V (I+l,-n))  is  nondecreasing  in  i 

c * 

'0  < i _ I)  Therefore  V^(ijn+1)  is  nondecreasing  in  i (0  _ i < I) 
since  the  minimum  of  two  nondecreasing  functions  is  also  nondecreasing, 
completing  the  mathematical  induction 

Consider  an  n-stage  problem  (l  < n < N).  For  0 • i • I, 

let 


expected  cost  of  not  repair 
for  a machine  in  state  i at 
the  beginning  followed  by 
the  best  rale 


expected  cost  of  repair  > 
for  s.  machine  in  state  i 
at  the  beginning  followed 
by  the  best  rule  / 


I 

■■  i)  - a t p.  . V (j|n-l) ) - tc(l)  + B -t  a(qi 
,j=0  1J  a 

+ (1-q)  Va(l+l,n-l))) 

Note  that  the  above  expression  also  holds  for  n 1 by  etting 


From  3,  Z*_q  Va(j|n-1)  is  nondecreasing  in  i (0  < i < I)  using 

Lemma  1.5  since  V (jjn-l)  is  nondecreasing  in  J (0  < j < I) . Other 

terms  are  independent  of  i.  Hence  fn(i)  is  nondecreasing  in  i 

. , * 

(0  < i < I)  for  any  1 < n < N„  Let  i be  the  smallest  i >0  such 
— — — — n n — 

that  f (i  ) > 0.  Note  that  by  the  assumption  that  A(l)  is  quite 
large,  fn(l)  > 0,  and  hence,  0 < i < I.  So  for  each  n (l  < n < N), 

■¥r 

there  exists  i such  that 
n 


A(i)  + a Z p V (j;n-l)  < C(i)  + B + a(qVjOjn-l)  + (l-q)V  (i+l; n-1) ) 


if  and  only  if  i < i^.  This  implies  that  at  the  start  of  each  n-stage 
problem  it  is  optimal  to  keep  the  machine  in  operation  if  its  state  i 
is  less  than  i^,  and  it  is  optimal  to  repair  it  if  i is  greater 


than  or  equal  to  in,  which  is  a control  limit  policy. 


□ 


Condition  1 says  that  the  material  cost  of  repairing  a machine 
must  increase  with  its  degree  of  deterioration,  while  2 means  the 
operating  cost  must  increase  more  than  the  increase  of  the  material 
cost.  3 assures  the  IFR  property  on  the  underlining  Markov  chain. 

The  above  result  can  be  easily  extended  to  the  total  expected 
a-discounted  cost  case. 


Theorem  2.2.  Assume  the  following  conditions  hold; 

1.  C(i)  is  nondecreasing  in  i for  0 < i < I. 

2.  A(i)  - C(i)  is  nondecreasing  in  i for  0 < i < I. 

3.  P1(-)  c Pi+1(0  for  0 < i < 1-1. 
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minimizing 


Then  there  exists  a stationary  control  limit  policy  R. 

1a 

the  total  expected  a-discounted  cost  of  the  model  where  i is  its 

Q! 

control  limit , 


Proof-  Theorem  1„1  shows  that  V^(i)  is  the  unique  solution  to 


V^li)  = mintA(i)  + a L p..Va(j),  C(i)  + B + CtCqV^O)  + (l-q)Va(l+l) ) ) , 

1 


0 < i < I 


(2,2) 


yi+1)  = B + a(qVa(0)  + fl-q)Va(l+l)) 


For  0 < i < I,  let 


fu)  = A( i)  + a L p.  .v  (j)  - t C ( i ) + B + a(qv  (0)  + (l-q)V  (1+1) 

j -0  J ^ u.  oc 


Then  Theorem  1.2  guarantees  that 


f(i)  - lim  f i)  , 
n ’ 

n -> 00 


0 < i < I 


Now  f(i)  is  nondecreasing  in  i (0  < i < I)  since  f (i)  is 

— - n 

nondecreasing  in  i for  each  n > 1,  which  has  been  proved  in  the 
last  theorem  Let  i be  the  smallest  i > 0 such  that  f(i)  > 0, 
Since  f(l)  > 0 from  the  assumption  that  A(I)  is  very  large, 

0 < i < I Then  for  0 < i < i , 

- a - -a 
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A ( i)  + a t p.,V  (3)  < C(i)  + B + a(qV(0)  + (l-q)V  (i+l) ) , 

j _Q  ^ vJ:  Of 


and  for  i < i < I, 
a — — 


A(i)  + a £ p. ,v  (j)  > C(i)  + B + a(qv  (0)  + (l-q)V  (i+l)) 
j=0  J a a 


By  the  uniqueness  of  the  solution  to  (2.2),  a stationary  control  limit 

policy  R is  optimal,  where  the  repair  decision  is  selected  if  and 

a 

only  if  i>iQl  □ 

Now  consider  the  long-run  average  cost  case.  First  a sufficient 
condition  under  which  Assumption  2 is  guaranteed  is  investigated. 

Definition.  A machine  is  said  to  eventually  fail  if  there  exists  n > 0 
such  that  p^  > 0 for  any  i (0  < i < 1-1)  where  p^  is  the 
probability  of  n-step  transition  from  state  i to  state  j. 


( 

I 


Suppose  a machine  eventually  fails.  Then  it  is  clear  that  no 
matter  which  stationary  policy  is  employed,  the  best  condition,  state  0, 
is  accessible  from  any  other  state  since  a failed  machine  must  be 
repaired,  and  after  the  repair,  it  regains  the  best  condition.  Therefore 
the  eventual  failure  property  assures  Assumption  2.  The  following 
theorem  then  holds. 


J 
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Theorem  2,3.  Assume  the  following  conditions  holds 

1.  C(i)  is  nondecreasing  in  i for  0 < i < I, 

2.  A(i)  - C(i)  is  nondecreasing  in  i for  0 < i < I. 

3.  PjO)  — Pi+1(-)  for  0 < i < 1-1. 

Also  assume  that  a machine  in  the  system  eventually  fails.  Then  there 

exists  a stationary  control  limit  policy  R ^ minimizing  the  long-run 

* 

expected  average  cost  of  the  model  where  i is  its  control  limit. 


Proof i For  each  a (0  < 0!  < 1),  there  exists  a stationary  control 

limit  policy  Ih  minimizing  the  total  a-discounted  cost  since  Theorem  2.2 

a 

holds.  Choose  a sequence  fc^}  (k  =0,1,...)  such  that  lim  ^0^  = 1 
and  such  that  for  any  c^,  the  same  control  limit  policy  is  optimal. 

It  is  possible  to  have  this  sequence  since  the  number  of  stationary 
control  limit  rules  is  finite,  F'ow 


VR.  „ U)  < V (j) 

i*,0!lE  R'°k 


for  any  R,  k and  j. 


Hence, 


) for  any  R,  k and  j . 


But  since  Theorem  1.4  holds  because  of  the  eventual  failure  property, 


Vp(d)  - vR(o)  =aiij(i-a)v  (o),  o < j < I, 


Therefore,  for  any  R and  j. 


VJ,'V0) 


< ilmd-alV  (0)  = V (0)  , V (o)  , 
k-«o  * K’ak  R R 


Therefore  the  stationary  control  limit  policy  R.*  minimizes  the  long-run 
average  cost,  where  i*  is  its  control  limit.  □ 

Conditions  1,  2 and  3 of  the  previous  theorems  are  known  to 
he  sufficient  conditions  for  the  optimality  of  control  limit  policies 
for  similar  systems  where  replacement  takes  the  place  of  repair.  That 
means  that,  as  far  as  sufficient  conditions  are  concerned,  the  model 
treated  in  this  section  and  the  corresponding  replacement  model  are 
essentially  the  same. 


j 

l 


2‘2'  Case  with  Arbitrary  Repair  Time  Distribution 

We  now  investigate  the  same  model  that  was  treated  in  the  last 
section  with  the  exception  that  the  class  of  repair  time  distributions 
is  now  expanded  from  that  of  geometric  distributions  to  include  any 
discrete  time  repair  distribution.  Let  T.  be  the  repair  time  of  a 
machine  in  statue  i.  Assume 


p(Ti = k)  = pk  - °» 

where 


k = 1,2, . . .5  0 < i < I 


00 


= 1. 


Note  that  the  repair  time  distribution  is  still  independent  of  the 
state  of  the  machine. 
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In  order  to  describe  the  model,  a countable  number  of  states 
labeled  1+1,  1+2,  . . . is  introduced  to  distinguish  the  length  of 
repair  times.  State  I+k  (k  = 1,2,...)  is  visited  when  a machine  is 
not  completely  repaired  after  k time  periods  from  the  start  of  its 
repair  work.  A repaired  machine  returns  to  state  0 as  before.  Let 
^k  the  probability  that  a machine  is  completely  repaired  in  the 
k-th  period  after  the  start  of  its  repair  work,  given  it  has  not  been 
completely  repaired  by  the  end  of  the  (k-l)-th  period.  That  is, 


P{  T . 


k 1 > k-1) 


t vk-1 
1 - h . P 
m--l  m 


k = 1,2, 


0 < i < I 


Let  Bi.k)  (k  = 1,2,...)  be  the  labor  cost  during  the  k-th 
period  after  the  beginning  of  repairing,. 


As  before,  V (i;n)  is  the  minimum  n period  a-discounted  cost 


starting  from  state  i,  while  V (i)  is  the  minimum  total  a-discounted 


cost  starting  from  state  i.  Then  recursively, 


mm 


VQ(i;l)  = min|'A(i),  C(i)+B}, 


Va(l+k;l)  = B(k)  , 


i = 0,1,  ...,I 


k = 1,2, .. . 


For  n > 1, 


Vdijn+l)  = min{A( i)  + ot  t p V (j;n),  C(i)  + B + a(qV  (0;n) 

JL  j J ^ i-  U 


+ (l-q^U+ljn))}, 


i - 0, 1, . . . , X 


Va(I+k:H-  , = B(k)  + Q!(qkVa(0;n)  + (l-^JV^I+k+ljn) ), 


k = 1,2,... 


L 


(2.3) 


We  suppose  that  there  exists  a finite  r.  iiber  M such  that 
! B(k) | < M for  any  k > 1„  Then  all  the  costs  are  hounded.  Now  if  the 
costs  are  bounded,  Theorems  1.1  through  1.4  are  known  to  hold  even  for 
the  case  where  the  state  space  is  countable  rather  than  finite.  Further, 
compare  the  third  equation  in  (2.3)  with  the  third  equation  in  (2.1). 
Nocice  that  they  are  essentially  the  same.  Therefore  the  same  method 
as  is  used  in  the  previous  section  can  be  applied  here  to  show  that 
ffi)  is  nondecreasing  in  i for  0 < i < I,  where 


f(i)  = Mi)  + a £ Fi(jVaU)  - (c(i)  + B + a(qiVa(0)  + (1-q^d+l) ) ) 

d — ^ 


As  a conclusion  we  have  the  following  theorem; 
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Theorem  2. 4.  Assume  the  following  conditions  hold: 

1.  C(i)  is  nondecreasing  in  i for  0 < i < I. 

2.  A(i)  -C(i)  is  nondecreasing  in  i for  0 < i < I. 

3.  PjOJCP  (•)  for  0 < i < 1-1. 

Then  there  exists  a stationary  control  limit  policy  R.  minimizing  the 

1a 

total  a-discounted  cost  of  the  model  with  arbitrary  repair  time  distri- 
bution, Furthermore,  if  a machine  in  the  system  eventually  fails,  there 
exists  a stationary  control  limit  policy  R ^ minimizing  the  long-run 
average  cost. 


2.3.  General  Case 

We  return  to  the  model  where  the  repair  time  has  a geometric 
distribution.  In  Section  2.1  the  repair  time  distribution  is  independent 
of  the  state  of  a machine  to  be  repaired.  In  this  section  the  case  where 
the  repair  time  distribution  may  depend  on  the  machine's  state  is  treated. 
Namely,  for  0 < i < I, 

P(T.  = j}  = qi(l-qi)'3"1,  i = 1,2,... 

The  other  generalization  is  the  dependence  of  the  labor  cost 
on  the  machine's  state.  Define 

B(i):  labor  cost  for  repairing  a machine  in  state  i per  period. 

The  rest  of  the  model  description  is  exactly  the  same  as  that  in 
Section  2. 1. 


2k 


1 


We  first  consider  the  total  a-discounted  cost  case  and  see  under 
what  conditions  some  control  limit  policy  optimizes  the  model.  In  order 
to  solve  this  problem,  we  augment  the  state  space  to  0,1, ...,I  and 
I + 1(0),  I + 1(1),  ...  , I + 1(1)  so  that  the  state  of  the  machine 
entering  the  repair  facility  can  be  memorized  during  its  repair  period. 
That  is,  if  a machine  in  state  i is  chosen  to  be  repaired,  it  goes  to 
the  repair  facility  reserved  for  the  state  I + l(i),  and  stays  there 
until  its  repair  is  completed.  The  number  of  time  periods  it  is  in 
state  I + l(i)  has  the  geometric  distribution  with  parameter  q^,  and 
the  labor  cost  B(i)  is  charged  per  period  while  it  is  in  state 
I + 1 (i ) . 

This  problem  again  can  be  formulated  as  a dynamic  programming 
problem  by  introducing  V^ijn)  as  being  the  minimum  n period  expected 
a-discounted  cost  starting  from  state  i (i  =0,1,..., I,  1+1(0),..., 
1+1(1)). 


Va(i;l)  = min{A(i),  C(i)+B(i)}, 


0 < i < I 


Va(l+l(i);l)  = B(i), 


0 < i < I 


For  n > 1, 


V (i;n+l)  = min{ A(i)  + a Z p V (j;n),  C(i)  + B(i) 
a J=0  ij  a 


(2.4) 


a(q.Va(0;n)  + (l-q.)Va(l+l(i) ;n) ) ) , 


0 < i < I 


Va(l+l(i),n+l)=B(l)  + a(qiVa(0;n)  + (l-q1)Va(l+l(i)  ;n) ) , 


0 < i < I . 


The  existence  of  a stationary  policy  minimizing  the  total 
a-discounted  cost  is  apparent  since  the  system  is  a Markov  decision 
process  with  discount  factor  0 < a < 1.  The  question  is  whether  or 
not  this  optimal  policy  can  be  a control  limit  policy.  What  makes  the 
analysis  more  difficult  than  that  previously  considered  is  the  fact  that 
the  right-hand  side  of  (2,U)  depends  heavily  on  state  i.  The  following 
lemma  is  useful  to  derive  the  conditions  under  which  a control  limit 
policy  minimizes  the  total  a-discounted  cost. 

Lemma  2.5.  Assume  the  following  conditions  hold: 

1.  C(i)  is  nondecreasing  in  i for  0 < i < I. 

2,  A(i)  is  nondecreasing  in  i for  0 < i < I. 

3.,  B(i)  is  nondecreasing  in  i for  0 < i < I. 

k.  q.  > q. , , > 0 for  0 < i < 1-1. 

5 P.(-)  C Pi+i(0  for  0 < i < 1-1. 

6,  C(0)  = 0. 

) 

Then, 

(a)  V (i;n)  is  nondecreasing  in  i (0  < i < I)  for  each  n > 1. 

a 

(b)  V (l+l( i) ;n)  is  nondecreasing  in  i (0  < i < I)  for  each  n > 1. 

a - “ 

i 

Proof"  First  notice  that  for  any  n > 1,  by  setting  V (i;0)  = 0 for 


i = 0,1, ...,I,I+1(0),... ,1+1(1), 


= B(0)  + 0:(aV  (0;n-l)  + (l-Q  )Vrv(l+l(0)  ;n- 1) ) 

v u:  0 JC 

i 

- mini  A (0)  +0  Z P0-iV  ( j ;n-l) , C(0)  + B(0) 

j=0  a 

+ a(Vat°5n'1)  • (l'qo)va(l+l(0)5n-1))} 

> B(0)  + a(qQVa(0;n-l)  + (l-^ )v  ( 1+1  (0 ) ;n-l) ) 

- IC(O)  r B(0)  + alq^tC^n-l)  + (l-q^V^I+lfO)  ;n-l) ) } 

= -C (0)  = 0,  by  6. 


The  proof  is  now  completed  by  mathematical  induction  applied 
simultaneously  on  the  two  statements  (a)  and  (b).  Both  (a)  and  (b) 
obviously  hold  for  n = 1,  from  (2.4)  and  conditions  1,  2 and  3.  Suppose 


they  hold  for  n = m > 1.  Then  for  0 < i < 1-1, 

ViV0>m)  + (1"(ii+i)va(I+1(ul);m)  - + (1-qi)va(i+i(i);m)) 

= qi+lV°'’m)  + (1"qi+1)(va(0;m)  +6i,m  + 6i,m  } 

~ (^iva(°i“)  + U-qi)(Va(0;m)  + 6^m)) 

* (,i  - WSi,m+  (1-Vl)sl,m  i **- 


where 


\m  VI+1U);m)  - Va(0;m) 

Va( 1+1(0 );m)  - V (0 ;m)  by  inductive  assumption 
> ® by  the  first  note  in  the  proof 


' 

i 
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8i,m  = VI+l(i+1);m)  - Va(l+l(i);m)  > 0 , 

by  inductive  assumption. 

Hence  q^V^Ojm)  + (l-q^)V_<;(l+l( i)  ;m)  is  nondecreasing  in  i for 
0 < i < I.  Because  of  this  fact,  and  by  3,  Va(l+l(i) ;m+l)  is  non- 
decreasing in  i for  0 < i < I.  Furthermore,  C(i)  + B(i)  + a(q^Va(0 ;m) 
+ (l-q^)Va(l+l(i) ;m) ) is  also  nondecreasing  in  i if  condition  1 is 

added.  Now  from  2 and  5 and  the  induction  hypothesis  on  (a)  for  n = m, 

I 

A(i)  + a ZJ=0  is  also  nondecreasing  in  i (0  < i < I) 

using  Lemma  1.5,  yielding  that  Va(i;m+1)  is  nondecreasing  in  i 
(0  < i < I).  Thus  both  (a)  and  (b)  hold  for  n = m+1,  completing  the 
mathematical  induction  and  the  proof  of  the  lemma.  □ 

Interpretation  of  the  conditions  in  the  previous  lemma  will  be 
given  later.  Using  the  results  of  Lemma  2.5,  sufficient  conditions  for 
the  optimality  of  a control  limit  policy  can  be  obtained. 

Theorem  2.6.  Assume  the  following  conditions  hold: 

1.  C(i)  is  nondecreasing  in  i for  0 < i < I. 

2.  B(i)  is  nondecreasing  in  i for  0 < i < I. 

3.  A(i)  - (C(i)  + B(i)/q^)  is  nondecreasing  in  i for  0 < i < I. 

U.  C(0)  = <">.  A(0)  > 0,  and  B(0)  > 0. 

5.  > Q1+1  > 0 for  0 < i < 1-1. 

6.  P^*)  pi+1(*)  for  0 < i < 1-1. 

Then  for  any  0 < a < 1,  there  exists  a stationary  control  limit  policy 
minimizing  the  total  expected  ct-discounted  cost  of  the  general  model 
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■ 


1 


Proof : First  notice  that  5 of  this  theorem  implies  2 of  Lemma  2.5  with 

the  assistance  of  conditions  1,  2 and  5.  Hence,  all  the  conditions 
of  Lemma  2.5  are  assumed  here.  Also  notice  that  1+  and  1 guarantee  that 
all  the  costs  are  nonnegative.  Therefore,  V (0)  > 0,  where  V (i)  is 
the  minimum  total  a-discounted  cost  starting  from  state  i. 

The  basic  idea  to  prove  this  theorem  is  similar  to  that  of 
Theorem  2.2,  and  it  is  enough  to  show  that  by  (2.4), 


I 

f(i)  = A(i)  + a L p.  .V  ( j ) 

i=o  J a 


- fC(i)rB(i)+a(q.VC[(0)  + (l-qi)Va(l-tl(i)))} 


is  nondecreasing  in  i for  0 < i < I.  From  Theorem  1.2  and  using  the 


result  of  the  previous  lemma,  V^Cjjn),  and  hence,  V (j)  are  nondecreasing 


in  j for  0 < j < I.  Therefore  by  condition  6,  n P.  .V  (j)  is 

J =U  1 J ot 

nondecreasing  in  i (0  < i < I)  using  Lemma  1.5.  So  it  is  sufficient 
to  show  that 


g(i)  - A(i)  - C(i)  - B(i)  - 0t(q.V  (0)  + (l-q. )V  <I+l(i) ) ) 

1 u 1 UC 


is  nondecreasing  in  i (0  < i < I).  Now  from  (2.4)  and  by  Theorem  1.1, 


V ( I * 1 ( i ) ) satisfies 


V (l+l(i) ) = B( i)  + a(q.V  (0)  + (l-q.)V  (l+l(i))),  0 < i < I. 

/ 1 (a  1 U. 


Therefore 


B(i)  + oq. V (0 ) 

Va(lrl(i))  = 


1 - a(l-q. ^ 


0 < i < I. 


Substituting  this  into  g(i)  yields 
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g(i)  = A(i) 
= A(i) 


n(t  \ B ( i ) + aq.V  (0) 

C(x)  - B(i)  - aq.V  <0  - a(l-q  ) *3. 

C(i)  - -iil 


°Va(o) 


l - a( l-q. , 


Hence, 


1 - a(l-qi)  " 1 _ a(i_q  ) 


g(i+l)  - g(i)  - [ (A(i+1)  - A(i))  _ (C(i-rl)  - c(i) ) ] 


r B(i+1) 

iW.  ' 

raqi+iva(°) 

aqiVa(0)  1 

l-a(l-qi)_ 

Ll-a(l-q.+1) 

l-Q!(l-q.)  1 

i - 

B(i+1) 

■ B(i)  1 

(qi-qi+i)  ad-a)V  (o) 

_l-a(l-qi+^T 

l-Q(l-q.T_ 

U-a(l-qi+1)) 

" d-a(l-q.  ) ) 

But  from  5,  and  by  the  fact  that  V (0)  >0 

Cr  ' - ’ 


(Vqi+i  )(1-a)3Vfy(°) 

W5 


u-a(i-q.+^ry(i_a(i_^7yj'  - 0 


Further,  if  we  let  b(i,a)  be  defined 


as 


m . 


b(i,a)  = — — ^ 


B(i) 


“ I-a(i_q4 ) 


then  for  0 < a < 1, 


Sb(i,q)  ^ (l-q.)  B(i) 

5 Q:  / , , " p ” —r  > 0 

(i-a(i-91+1)J2  (l-aii-,^)2  ~ 


b"““  °f  C°naItl°"s  2 “a  5.  Hence,  for  0 < a < 1,  and  for  each 
0 < i < 1-1, 


b(i,a)  < b(i, l)  = Mifl)  _ B(i) 


i+1 


qi  ’ 
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Therefore  for  0 < i < 1-1,  by  using  condition  3, 


g(i+l)  - g(i)  > (A ( i+1)  - A ( i ) ) - (c(i+l)  - C(i) ) - ( > 0 

\ qi+l  qi/“ 

which  completes  the  proof.  □ 

Interpretation  of  each  condition  in  the  above  theorem  is  as 
follows.  1 and  2 indicate  that  both  material  and  labor  costs  increase 
as  the  condition  of  the  machine  to  be  repaired  gets  worse.  4 assures 
the  nonnegativity  of  all  costs  and  that  no  material  cost  is  incurred 
if  the  machine  is  in  the  best  condition.  5 means  that  the  worse  the 
condition  the  machine  is  in,  the  longer  the  time  reauired  to  repair 
it  will  be.  The  IFR  property  is  guaranteed  by  6.  Now  consider  condition  3. 
Note  that  B(i)/o^  is  interpreted  as  the  expected  labor  cost  for  a single 
repair  of  a machine  in  state  i.  Hence  3 says  the  operating  cost  must 


increase  more  than  the  increase  of  the  combined  material  and  expected 


labor  costs  for  repairing  a machine. 


The  long-run  average  cost  version  of  the  previous  theorem 


can  be  obtained  just  as  in  Theorem  2.3,  and  stated  as  given  below: 

Theorem  2.7 . Assume  all  the  conditions  in  Theorem  2.6  hold  and  further- 
more suppose  the  machine  in  the  system  eventually  fails.  Then  there 
exists  a stationary  control  limit  policy  minimizing  the  long-run 
expected  average  cost  of  the  general  model. 
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CHAPTER  3 

MAINTENANCE  MODELS  WITH  SPAKE  UNITS 

In  Chapter  2 several  models  without  spare  units  are  studied. 

In  this  chapter  the  previous  models  are  generalized  in  the  direction 
to  include  systems  with  a finite  number  of  spare  units.  This  generali- 
zation seems  appropriate  since  spare  units  are  often  introduced  in 
maintenance  problems  to  avoid  the  system's  failure.  Obtaining  sufficient 
conditions  for  the  optimality  of  some  kind  of  control  limit  policy  is 
of  interest.  Some  extensions  and  modifications  are  also  presented. 


•. 1.  The  Model 


A general  machine  repaii  model  with  spare  units  is  schematically 
st.own  in  Fig  3 1,  There  is  an  operating  machine  and  S (S  > 1^  spare 

machines  in  the  system.  The  operating 
machine  deteriorates  as  time  goes  on 
and  is  classified  as  being  in  one  of  I+1 
states  as  before  If  there  is  an 
operating  machine,  two  choices  are 
available  by  a decision  maker  at  each 
period:  to  let  it  keep  operating,  or 

to  send  it  to  a repair  shop 
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Figure  3 1»  A General  Machine 
Repair  System 
With  Spare  Units. 


As  before  p. . is  the  one  step  transition  probability  from  state  i 
to  state  3 when  the  former  decision  is  selected.  When  the  latter  is 
selected,  the  operating  machine  is  immediately  sent  to  a repair  shop 
and  is  replaced  by  one  of  the  spare  machines,  if  any.  The  new  operating 
machine  begins  to  operate  at  the  end  of  the  period  in  its  best  condition. 
The  repair  time  T of  a machine  in  state  i is  a random  variable  having 
a geometric  distribution  with  parameter  q^,  i.e., 

P{Ti  = j)  = qi(l-qi)'3"1,  j = 1,2,...  . 

Machines  in  the  repair  shop  are  independently  repaired.  If  all  the 
machines  are  in  the  repair  shop,  no  machine  is  available  and  the  system 
fails.  In  this  case  there  is  no  option  available  but  to  wait  until  one 
of  the  machines  is  completely  repaired.  A penalty  cost  P is  assessed 
per  period  during  the  system's  failure.  Other  costs  are  the  operating 
cost  A(i),  material  cost  C(i)  and  labor  cost  B(i),  which  are  the 
same  as  the  ones  defined  in  Chapter  2.  The  ob3ective  is  to  find  a 
repair  policy  minimizing  the  total  expected  a-discounted  cost. 

In  order  to  describe  the  model  we  need  to  specify  the  number  of 
machines  in  the  repair  shop  coming  from  the  i-th  operating  condition 
for  each  0 < i < I,  as  well  as  the  state  of  the  operating  machine. 

The  state  of  the  system  is  then  represented  by 


Xt  - (x°, 


) - ( i,  Sq,  Sp  . 


SI} 


when  there  is  an  operating  machine  at  the  t-th  period  whose  operating 
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condition  is  i,  and  (0  < k < I)  machines  are  being  repaired 

which  have  been  in  the  k-th  operating  condition  when  the  decision  to 
repair  is  made.  Here, 

I 

0 < 1 < I,  0<  Z sw<S,  s > 0 for  0 < k < I. 

k=0  k ~ ~ 

For  notational  convenience,  we  write 

~ Sq,  s^,  . . . , Sj) 

when  ^k=0  sk  = S + 1 and  sk  - 0 for  0 - k 1 x>  i-e-»  when  a11  the 

machines  are  under  repair. 

For  simplifying  the  repetitive  use  of  the  following  sets,  we  let 

oP  = C i | 0 < i < I,  i: integer} 

I 

J - { (sn, . . . , s_) | £ s < m,  s.  (0  < k < I) : nonnegative  integer} 

~ k=0  k - - 

= { (0,  s , . . . , sT)  | £ s = m,  s (0  < k < I) : nonnegative  integer}, 

u 1 k=0  k k-- 

This  Markov  decision  model  has  the  following  dynamic  programming 

formulation.  Let  V (i, sQ, . . . , s^;n)  be  the  minimum  expected  n period 

a-discounted  cost  starting  from  state  (i,  s_, . . . , sT) . Let  be 

0 1 ss' 

the  probability  that  s'  out  of  s machines  are  still  under  repair 
at  the  end  of  that  period  given  s machines  are  in  the  repair  shop 
coming  from  the  j-th  operating  condition  at  the  beginning  of  a period. 
Because  of  the  independence  of  each  machine  under  repair, 


i \ s\  s -s' 

) (1-qJ  J q,J  0 

s'.  / J J 

J y 


0 < j < I , 


since  repa:  r time  T.  has  the  geometric  distribution  with  parameter 

J 

q-  Then  by  setting  VQ(i, s0> . . . , s^O)  = 0 for  any  feasible  (i,  sQ, . . . , s ) , 
V^(i, Sq, . . . , s^; n)  (n  > 1)  satisfies  a set  of  recursive  equations: 


Va(i'V---’sIjn) 


I s0  SI 


= min{A(i)  + Z B(j)s  + a Z Z ...  Z p , 
j=0  0 i'=0  s^=0  s j=0  11 


qsQs^  •••  V1,,s6,***,si;n“1^ 


I so  si+1  SI 

C(i)  + Z B ( j ) s . + B(i)  + a Z ...  Z ...  Z 

J=0  J Sq=0  8^=0  sj=0 

qVo  q®i+1’si  qsx4  V0,so,,”,si5n'1)) 


for  i £ M , (s0,...,Sl)  € J 


(3.1) 


^a(°>  sq>  • • • > si> n) 

I so 

= P + Z B(j)s . + a Z . 


for  (i, Sq, . . . , Sj)  € J q 


Z q(0),  . 

s j=0  S0S0 


qiIiiTa(0'-0’-'*i>n-1> 


Let 


B (i,s0> 


.STjnj 


) = 


E . 
so=0 


i 

E q 

1=0 


(0) 

soso 


. q^1 ^ V (i, s' , 
s si  a 0 

J J. 


■ > sj;  n) 


(5.2) 


Then  (5.1)  can  be  simplified  to 


vc^i,so,"*,srn) 

i i 

= min(A(i)  + E B(j)s  +a  E p R (i ’ , s0, . . . , s ; n-1) , 
j=0  3 i ' =0  1 

I 

c(i)  + E B(j)s+B(i)  + aR  (0,  s_, . . . , s. +1, . . . , s ; n-1) ] 

j=0  3 a 

for  i£j,  (aQ, . . . ,Sj)  £ > (5.5) 

I 

Va(°,S0’ ' - ,,sl’n^  ~ P + ^ B(j)s_.  + Q&atO,s0,...,sI;n-l) 

for  (i, sQ, . . ..Sj)  1 . 


A simple  control  limit  policy  defined  in  Chapter  2 can  not  be 
expected  to  be  an  optimal  policy  since  the  state  of  the  system  depends 
on  the  number  of  machines  in  the  repair  shop  as  well  as  the  condition 
of  an  operating  machine.  The  following  type  of  control  limit  policies 
is  now  a suitable  candidate  for  the  class  of  policies  satisfied  by  an 
optimal  policy. 
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Definition.  An  i.  control  limit  policy  is  a nonrandomized  policy 
where  there  is  an  i for  each  feasible  s = (Sq,...^)  and  for  each 
period  n > 1,  say  ig  called  the  control  limit,  such  that  for  all 

with  i < i the  decision  at  period  n is  to  keep  the 


( i f Sq  * • • • > ^ J I 


machine  in  operation  and  for  all  (i, sn, . . . , sT)  with  i > i the 

u I - s,n 

decision  at  period  n is  to  repair  it.  It  is  called  stationary  if  i 

s,n 

is  constant  in  n for  each  s. 


We  now  investigate  the  conditions  under  which  the  existence 
of  a stationary  i control  limit  policy  minimizing  the  total  ce-dis- 
counted  cost  is  assured.  We  first  prove  several  lemmas. 

Lemma  3.1.  If  V(i)  = min(V1(i),  V2(i))  for  0 < i < I and  if 

vk(i)  ~ Vk(i-1)  > M for  k = 1,2  and  1 < i < I,  then  V(i)  -V(i-l)  > M, 

1 < i < I. 

Proof:  For  1 < i < I, 

V(i)  -V(i-l)  >min(V1(i),V2(i))  -min(V1(i)  - M,  V2(i)  - M)  =M  . □ 

Lemma  3-2.  If  V(i)  = min(V1(i),V0(i) ) for  0<i<I  and  if 

vk(i)  " vk(i-1)  < M for  k = 1,2  and  1 < i < I,  then  V(i)  -V(i-1)<  M, 

1 < i < i:. 

Proof:  For  1 < i < I, 

V(i)  - V(i-l)  <min(V1(i),V2(i))  - min(V1(i)  -M,  V2(i)-M)=M  . □ 
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Lemma 


IaI-  For  i€«J?,  (s0 sz)  6/3’1  and  for 


( sq>  • • • t s j ) 


;'s, 

O’ 


Rc^1,sO'*,,,sk+1'* 


■ > sr;  n) 


= (i-O  £ .. 
*0-° 


°k 

I . 

s;=o 

k 


£ q(0),  ... 

Sj=0  S0S0 


i™. 

sisi 


ij  s0’  ’ * ’ ' sv+1>  si  j n)  “ Va(i>sQ, . . . , sk, . . . , sj;n) ) 


for  0 < k < I and  for  n > 1. 


Proofs  For  i £ J.  , (sQ, . . . , sy)  <=  iS_1  and  for  (i,  sn, . . . , sT)  € J 
using  (3.2), 

S0’ ’ ' ’ ’ sk+^’ ' ’ ' ’ sI’n)  ~ Ea^'sn',,,'V",>sT}n) 

SI 


O’ 


Sk-1  sk+l 


cr  ’ o’ ' ' ' ’"V  ’ “i’ 

s. 


Z ...  Z "I  ...  Z q,0),...q(l!-1),  ,<k+1>, 
so"°  si-l“°  sk+i"°  si=°  S°S°  sk-lsk-l  sk+lsk+l  slsi 


rsk+1 


I-  k 


s^=0  k k 


and  n. 


Now  use  the  following  relation.  For  nonnegative  integers 


'n-1'  V 11  n ’ 


Note  that  the  above  equation  holds  for  n = 0 and  for  n > m by  setting 
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(m  + k)  =0  for  k>* 


(q)  = 1,  (_J)  « 0 . 


^ qs  •♦■1  s,Va^' 8o*  * • •j8!'1 

i'=0  Vi,8k a 0 1 


,^0  qsk8i  VQt^1' S6* 


^T-1  /^Sfc+1  \ Si  SV+1’SW 

Vs-  Z1'^  q*  v^86— si^) 

k k 

“k/8k\  K 

- . „. ) (1-V  \ 


w** 


■k-°  \ "i-1- 


Sk  8k+1_8k 

qk  Vgj(  i»  *q»  • • • > 8 j j n) 


V1  / 8k 

+ E f 

8k=°  \ 8k . 


sk  r\ 


ii  8k+1-sk 


(i-V  qkK  Va(i,s(i,...,ai,n) 


8>tbV  sk  ^1_qk^  qk  Vo/i,86',,*'8i;n) 


Sk  8k"8k 


8k/8k  \ s’+l  a. -s' 

wA  a.  / 1_qk^  qk  va(i»so»**-»s^+l>...»sjjn) 


8i=°  V 8k 


s.'  S.  -s.' 


+ (V\8k  Z1'^  k ^ k Va(1'86 

(1_qk)  [8^  qvi  Vl»86»"-»8i+1^-*>8^n) 

" s£*>  q*k8k  Vl'86'-'8i>n>]  * 
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so'  ’ ‘ * ’ sk'  sk+i+^' ' ‘ ’ si’ n'^  “ Ra( ^ so*  * • * * sk+^'  sk+l-’ 1 sx> 


Ra^'  so’  ’ * ’’  sk’  sk+i+1>  " ’ ’ si>"n)  ~ Ra^,s0’ — >sx»n^ 

S0>  ” ” ’ > Sl<-+^->  • • " > ST>  ” Rq,(  SQ>  • • • ) S j)  Q)  ) 


-d-w  L - £<••••  <. 

Sq=0  s^=0  0 0 II 


• »Sk+l+1»  1 • • > ST^‘ n)  " Vr.(i»SOJ”l>Ski.l) 


p0  SI 


cr  ’ o’  * * k+i* 


• • y s j>  *0 ) 


(l-qk)  E ...  E q<°>  ...  q(l), 

Sq =0  s j=0  0S0  SISI 


■ - Vrt(i,s^...,8;,...,s;jn)) 


cr  ’ o’  * k’  ■ * * ’ i’ 


(l-qk)  E ...  E q<°>  ...  q(l), 

k Sq=0  s’=0  S0S0  sIsi 


^Va^1,s6, ',sk-l+1,  ”»^sx’n-  " Vi'so',"'sv+1>>..»sTJn)) 


s0  si 

+ (qk-Vi>  E...  E q(0), 

k k 1 Sq=0  s’=0  S0S0 


V (0  (I 

L q ....  q , 

^<3  <3  ' <3  I 


■ (yi,S’,...,si+1+l,...,si!n)  - V0(i,s',...,si+1,...,5i,„)). 


So  by  condition  1,  if  (a)  and  (c)  hold  then  the  last  expression  becomes 
nonnegative,  and  hence  (b)  holds. 

What  is  left  is  to  prove  (a)  and  (c),  and  this  is  done  by 
mathematical  induction.  For  n = 1,  (a)  is  obvious  because  of  3 and 
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(c)  is  also  clear  by  2 and  3.  Suppose  they  both  hold  for  n = m-1  > 1. 
Then  by  the  previous  argument,  (a)  through  (d)  hold  for  n = m-1. 
Consider  the  case  for  n = m.  Take  (a)  case  first.  For  (i,  s^, . . . , s^.)  € 

v^°’s0>  •••>  Sj^_j_^+1,  . . . , s j,  m)  - (0,  Sq,  . . . , sk+l, , . . , s m; 

I 

= P + _Z  B(j)Sj  + B(k+1)  + aiyo,s0,  ...,sk+1+l,  ...jSjjm-l) 

I 

- (P  + E B(j)sj  + B(k)  + aRa(°>s0»...»sk+l, . ..jSjjm-l)) 

>0, 

from  3 and  by  inductive  assumption  on  (b)  for  n = m-1. 

For  i £ , (Sq,...,Sj)  1 jJl S \ we  compare  the  corresponding 

values  term  by  term  using  (3.3). 

For  notational  convenience  if 

V(i)  = min(V1(i),  V2(i),  ...  , Vm(i)}  , 

in  order  to  represent  the  k-th  tern  V (i),  we  write  (V(i)), 

K K"  bil 

(1  < k < m)  throughout  this  paper. 


(Va(i,  sQ, . . . , sk+1+l, . . . , Sjim)  )1.st'(VaU,  sQ, . . . , sk+l, . . . , Sjjm)  >L_st 
I I 

= A(i)  + E B(j)s.  + B(k+1)  + a £ p.  . ,R  (i' ,s  , . . .,s  +1, . . . ,sT>m-l) 

j=0  J i’=o  11  a u K+i  1 

I I 

- (A(i)+ £ B(J)S, +B(k) +a  E p,.,R(i' 

d=o  J i’=o  1 a 

> 0, 

1+2 


> Sq, • . . , sk+l, • . . , Sjj m-1) ) 


o d* 


from  3 and  by  inductive  assumption  on  (b)  for  n = m-1.  Similarly, 


we  have 


' * •^k+l’^1’ ' • ^2-nd  ” <'Vc/1,s0,  * • • • *'sI»m)  >2-nd 


Therefore,  by  Lemma  3.1,  for  i £ , (Sq,...^)  £ 


i,  Sq,  . . . , sk^_^+l, . ..,Sjjm)  — V„(  i,  s^, « • • , s^+1,  ...,sT^m)  > 0 


or  o , k 


Hence  (a)  holds  for  n = m.  Consider  (c)  case.  For  i £ , 


(s0,...,Sl)  by  using  the  similar  technique. 


'Vc^ i’  s0’  * * * ’ sk+'^  ’ ’ • ’ SI;  m^i-st  " 'Va^ 1>  s0’  * ’ * ’ sk’  * • * > si»  m^i-st 


A(i)  + E B(o)s..  + B(k)  + a E Pii,Ra(i',s0,...,sk+l,...,sIjm-l) 


- (A(i)  + E B(j)s  + a E p R (i',s  ,...,s  ,...,s  jm-l)) 
j=0  J i'=0  -1  a u K x 


B(k)  + a E Piit  (p.a(i' , sQ, . . .,  sk+l, . . .,sI;m-l) 

“ B^( x > * * * * sj,m-l) ) 


from  3 and  by  inductive  assumption  on  (d)  for  n = m-1.  In  a similar 
manner,  we  have 
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i > Sq>  ■ • . t sk+l,  ^0’  * *#,^k’  * ^2-r 


which  gives  us  that  for  i £ ^ , (sQ, . . . , s^.)  £ 


V^( i,  Sq>  . . . ^ s^+-l, . . . , St^ m)  > V_^( i,  sn, . . . , . . . , sTj m) 


For  (i,s0,...,sI)  £^  ®, 


^0’  * * * ’ * * ’ * “ ^^(0,  StV  ’ ' * > ’ ■ * * 


a'  ’ o’ 


>P+  L B(j)s^  + B(k)  + aRa(0,s0,...,sk+l,...,sIjm-l) 

0-^0 

I 

- (C(0)  + L B(j)Sj  + B(0)  + °®a(0,s0+l, ...,sk, ...jSjjm-l)) 

= P - C(0)  + B(k)  - B(O) 

+ ^0*  * * ’ ’ Sk+^,  * • • > ® j»  m-l)  -R^(0,  Sq+1,  . . . , sfc, . . . , s ^jm-l) ) 


by  2 and  3 and  inductive  assumption  on  (b)  for  n = m-l. 

Hence  for  all  cases  (c)  holds  for  n = m.  Thus  both  (a)  and  (c)  hold 
for  n = m,  completing  the  mathematical  induction  and  the  proof.  □ 


Lemma  3.3.  Assume  the  following  conditions  hold: 

1.  0 < qk+1  < qk  < 1 for  0 < k < 1-1. 

2.  P > C(0). 

3.  0 < B(k)  < B(k+1)  for  0 < k < 1-1. 


kh 


4. 


Pi(-)  C Pi+1(.)  for  0 < i < 

5.  A(i)  is  nondecreasing  in  i 

6.  C(i)  is  nondecreasing  in  i 
Then  both  va  (i, sQ, . . . , Sjj n)  and 
in  i (0  < i < I)  for  (sQ, ...,s 


1-1  where  P (j)  = Z p . 

1 k<j  lk 

for  0 < i < I. 
for  0 < i < I. 

Ra(i> sq) • • • , SjJ n)  are  nondecreasing 
j)  £ S and  n > 1. 


Proof:  First  note  that  if  V^(i, s^, . . . , s^jn)  is  nondecreasing  in  i 

(0  < i < I),  then  by  the  definition  of  (i, sQ, . . . , n) , which  is 
equation  (3.2),  Ra(i,  sQ,  . . . , s^.;  n)  is  also  nondecreasing  in  i (0  < i < I). 
Now  the  proof  is  by  mathematical  induction.  For  n = 1, 


Va^i’  s0’  '*  ,,si'  -1)  = minfA(i) 


I I 

£ E(d)s.,  C(i)  + Z B(j)s,  + B(i) ) 
0=0  J j=0  J 


is  clearly  nondecreasing  in  i by  3,  5 and  6.  Suppose 

s0’  * * " * ^ 2)  nondecreasing  in  i (0  < i < I). 

For  (sq, ...,Sj)  c x/  , 


^VC^1,S0'  • ‘ ^i_st 


I I 

= A(i)  + Z B(J)S  + a Z p R (i’,s  ,sT,m-l), 

0=0  J i ’ =o  11  a u 1 


As  Ra^'  ,30’ * " ’ is  nondecreasing  in  i'  (0  < i'  < I),  by 

k and  using  Lemma  1.5,  Zj1=<)  Pu,  RQ(i’ , sQ, . . . , Sjjm-l)  is  nondecreasing 
in  i (0  < i < I).  Further  A(i)  is  nondecreasing  in  i (0  < i < I), 
by  5.  Hence  (V^/i, sQ, . . . , s^jm) )1_gt  is  nondecreasing  in  i (0  < i < I). 
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Also, 


7T- 


''Vi,S0’‘-',SI;m)  *2-nd 

I 

= C(i)  + B( j)s  + B(i)  + a*  (0,so,...,s4+l,...,sT;m-l)  . 

i =0  ^ jc  v i 1 

Notice  that  Lemma  3.4  holds  because  of  conditions  1,  2 and  3.  So 
Ra(0, Sq, • . . , Si+1, . . . , si;m-l)  is  nondecreasing  in  i (0<i<I)by 
(b)  of  Lemma  3.4.  Therefore  va(i,  B(), . . Sjjm)  > is  also  non- 
decreasing in  i (0  < i < I)  since  both  C(i)  and  B(i)  are  non- 
decreasing in  i by  3 and  6.  Thus  va(i, sQ, . . . , s ,m)  is  non- 
decreasing in  i (0  < i < I),  completing  the  induction  and  the  lemma.  □ 


The  property  that  v (i,  sQ, . . .jS^n)  is  nondecreasing  in  i 
(0  < i < I)  is  necessary  but  monotonicity  is  not  enough  to  show  that 
an  optimal  policy  is  assured  to  be  of  an  i control  limit  form.  In 

order  to  obtain  other  conditions,  bounds  on  V (i,  sn, . . . , s.+l, . . . , sT,  n) 

Ql  0 k X 

“ V ^ v ^ • • • r sk,  •••>Sj,n)  an(  1 V_^  ( i , s^  ,...,s^,  s^_^+l , ■ ■ . , s j n) 

” V^(i,  s0,...,sk+l,sk+1>...,sI>n)  must  be  calculated.  Lower  bounds  on 
them  are  given  in  Lemma  3.4.  Upper  bounds  are  shown  in  the  next  two 
lemmas . 


Lemma  3. L.  Assume  conditions  1,  3,  4,  5 and  6 of  Lemma  3.5  hold  and 
furthermore  assume  the  following  condition  holds: 

2.  P > C(0)  + B(0). 

Then  for  0 < k < I and  n > 1, 
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Va^1,sO,’**,sk+1','''sl;n)  " Vrv^i>so»-"»s  ;n) 


a'*’ “Q> . - ■> , o , . . . , a jnj  < U 
u u k I - a,  n 


for  i £ 


■S-l 


JL  , (s0,...,sx)  £ J~  - and  for  (i, sQ,  . . . , sj  € J * where 


U 


k (1-a  (l-q Jn)G(k) 


a,  n 


l - all -a  )■ 

k 


and 


G(k',  = P + B{  k)  - mint  A (0),  C(0)  + B(0)} 


Proof,  Mathematical  induction  is  again  used.  For  n = 1,  if  i £ J? 
an-j  if  (sq,..,,Sj)  £ jJ  ® \ then 

va(1'V*“»sk+1»*-*»sr1)  " va(i»V-*^V”"sI;i)  = B(k)  . 

For  (i,s0,...,Sl)  £ J*, 


Vo:(0’  50' ” ’ ‘ ’ sk"^1, ' ’ ° ’ SIJ 1)  - V°,s0,.,,,st,,„,,sT5l) 


P + L B ( ,j ) s ^ + B(k)  - min(A(0)  + £ 


3=0 


B ( 3 ) s , C ( 0 ) + Z B ( j ) s . + B (0 ) } 
3=0  J j^)  J 


= p + B(k)  * minfA(O),  C(0)  + B(0) J - G(k), 


Now, 


^ < k ' - P + B(k)  - mint  A (0 ) , C(0)  + B(0)j 
> P + B(k)  - (C(0)  + B(0) ) > B(k) , by 


Hence, 


V1'80,***»V1*-**»srD  ■ V1,8o»**,»Bk,"*»8i!l)  - G(k)  = v*  • 

(X,  1 
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1 


So  the  hypothesis  is  true  for  n = 1.  Suppose  it  is  true  for 
n = m-1  > 1,  and  consider  the  case  for  n = m.  Now  from  Lemma  3.3, 


S0’ • • • » ^ - ®ry( i>  so>  • • • ) s^j . • . , sTj m-l) 


"V 


cr  ' o’  • • • > • • • > 


so  si 


-d-O  2...  I qf°>  ...  a., 


(I) 


k Sq=0  s j-0  *S0S6  ‘SISI 


(va(i>  s0>  • • • » sk+l>  • • • ) si.;m“l)  - Va(i,  sQ, . . . , sk,  . . . , Sj.jm-1)) 


£ d-O  < by  inductive  assumption. 


For  (s0, . . .,sr)  £ JS'\ 


V 1 * * ’ sk+1’  • ' * > sIjm^  \-st  " ^c/1'  V ‘ ‘ * ' V ‘ * ' • sl;r^  ^l- 


st 


= S(i)  ♦ B(J)Sj  ♦ B(k)  + Pu,Bo(i',s0,...,Sk+l,...,Sl;m-l) 


- (A(i)  + B(d)Sj  + d Pii,Ra(i',s0,...,sk,...,sI;m-l)) 


d=0 


< B(k)  + d Z p (1-q  )Uk  = B(k)  + a(l-q,  )Uk  . 

i'=0  11  k a>ni-l  k a,  m-l 


In  a similar  manner,  we  can  get  the  same  upper  bound  on  the  difference 


of  the  corresponding  second  terms.  Hence  for  (s_, ...,s  ) £ ^ S_1, 


i £ , using  Lemma  3.2, 
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Va'i' so,  ‘ sk.+1,  * " ’ si'm^  " va^i»so>'”-’s^ 


< BOO  + • 


For  (i,s0#...,8l)  € 


Va^0'  sO*  " ' ' sk+1»  • ' • ’ sT,m^  “ va^°>  sq>  • • •>  sk>  • • • > SjJ111) 


= P + E_  B(j)s.  + B(k)  + aRa(O,s0,...,sk+l,...,sTjm-l) 


I I 

min(A(0)  + E B(J)s  + a E p B (i* , s , . . s_;m-l), 
0=0  J i'=Ouia  u 1 

I 

C(O)  + E B(d)s  + B(O)  + OR (0,s  +1, ...,sTjm-l)) 

j =Q  LX  U 1 


< P + B(k)  + aRa(0,s0,...,sv+l, ...,sT,m-l) 


- min(A(0)  + aR^CO, sQ,  . . . , Sjjm-l) , 


C(0)  + B(0)  + CXRa(0,s0,...,sI;m-l))  by  Lemmas  l.h  and  3.5 


= G(k)  + a(Ra(0,s0,...,sk+l,...,sT;m-l)  - Rq(0, sQ, . . . , sj m-1) ) 


< G(k)  + a(l-qk)U^  m_1  , by  inductive  assumption. 


As  G(k)  > B(k)  by  2,  for  all  cases, 

'*'sk+1'  •••>si5tn)  “ Va(  i>  s0>  • • • > sk> . . . , Sj.;  m) 

< G(k)  + a(l-q.  )Uk  . = U* 

- k a, m-1  a, m 

which  completes  the  induction  and  the  proof  of  the  lemma. 


Let 


I 

I I 


Uk  =■  lim  Uk  = 

a n^oo  a>n  1 - <*0--v 


r,k 


Then  as  U^  is  nondecreasing  in  n for  any  a and  k,  under  the 
same  conditions  as  in  Lemma  5.6,  we  have 

Va^1,so'‘*‘,sk+1'*,,,si;n)  " Va(i,s0' ‘“'V  •••  ^sI5n)<U 

Using  the  above  lemma,  we  can  derive  an  upper  bound  on 

Va^i>  s0*  ‘ - • ’ sk+l+]w  • * ' * si»n-  " Vc/ij  s0’  ‘ * • * sk+lj’  ‘ ‘ '»si^n)  which  is  in 
turn  used  to  obtain  the  main  theorem. 

Lgmma  5.7.  If  all  the  conditions  in  Lemma  5.6  hold,  then  for  i e-y1-, 
(s0,...,S;L)  e and  for  (i,s  ,...,s  ) e J* 


Va' L*  V • • * » V sk+l+1’  • ■ • ’ si5  n)  " va^>  s0’  * • * ’ sk+1>  sk+i>  • • • >sx->  n) 

° < k < 1-1,  n > 1 , 


M*  ' “k 


where 


< NT  < m , 
- a,  n - a ’ 


.k 


Q!,n  1-aTTqTT  lB^kfx)  " Blk)  + a^qk"qk+l)US  1' 


and 


a 


n -*® 


k B(k+1)  - B'k)  + “(^-qk+^u; 

4 - 

a,  n 


Mk  = lim  MK  = — — - .r^k 


k+1 


l-a(l-q  ) 


i 4 
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Q Pf 


}. 

i. 


Proof.  Mathematics’!  induction  is  applied.  For  n ~ 1 if 

U,s0’  • • • « ji  q, 


V0’  V • • • ’ V sk-if1'  • ■ ■ » sr 1)  ‘ V0’  V - • • > V1’  sk+r  • • • > SXJ  i) 


- P + J-  E(j'is_  + B(k+1)  - (P  + E B(j)s.  + B(k) ) 


= B(krl)  - Bik- 


< B(k+1)  - B(k)  + a(q  -q  )Uk'L  = by  1 

k k hiy  a a,  l y 

Similarly  for  i e JL  and  (sQ,...,s  ) e £S~1, 


or  ’ 0”"’  k ' k+1’ 


- B(k+1)  - B(k)  < 

— OL,  1 


So  the  hypothesis  holds  for  n --  1.  Suppose  that  it  holds  for 
n - m-1  > 1.  Then 


V ij  s0  ’ ' " ' ’ sk-lf1’  • • ■ » sr  m~E)  R0(i,  sQ, . . . , sk+l, . . . , Sj) m-1) 


iRavl,so,’,-,sktifl'-"’>si'm*i)  - Ra(i,s0' ‘ ’ ’’ sI»m"l) ) 


'Rctl i’ s0’ • • - ,skfl’ * * • >si?m-l.  ” Ra^i> so’ ’ ‘ * ’ sjJm-l) ) 


(1-qk+l}  z 

s’=° 


Z q(0) , 

sj=0  !0S0 


1(I) 

sisi 


^Va^1,s0’ ' * * ’sk+l+1’  * ' ' va^'  so>  • • • 'sk+i»  • • ->sj 


S0  SI 


- (1-Qk)  I ...  I q<°>.  ...  Cl 

Sq=0  s’=0  S0S0  ^ISI 


S0>  S]t  sj5  m-l ) " V^(i,  sq>  • • •>  s^., . . . , sj)m- 

by  Lerame  3.3 


0 

(l-q.)  Z . 

s’=° 


2 q(0), 
si=o  soso 


1(I) 

SISI 


^Va^i;,S6-’  ‘ ’ sk+l+‘L;’  * * “ va(i.>  sq>  • • •>  sk+1-’  • • • ’ si 


+ (q.-q^)  Z ...  Z q[0)  ~(I) 


k k+1  3^=0  s j=0  ^sos6 


SISI 


(va(i>  s0>  • . .,s^+1't-q, . . . ■va'i»so»  ••  •>si+i»  • 

s s 

< (l-qk)  Z ...  Z q(0l  ...  q(l),  ^ n 

k ~ Sq=0  s0s0  sIsI  a>m_1 


•’si 


3^0 


+ (\-w>  k •••  £ q?l  •••  <£+1 > 


so--° 


s'  S0S0 


a. 


by  Lemma  3.6  and  inductive  assumption 
(1-qk^Ia,m-l  + ;qk-qk+l)Ua  * 


5& 


» m-l) ) 


D), 


’>  m-l) ) 


5 m-l)) 


'by  comparing 


sT;m)  term 


by  term  and  using  Lemma  3,2,  we  can  similarly  show  that 


which  completes  the  mathematical  induction  for  V 


is  nondecreasing  in  n for  each  a and  k 


ince 


In  the  proof  of  Lemma  3-7>  we  also  showed  that  if  all  the 


conditions  in  Lemma  3.7  hold,  then  for  i € and  (sQ, . ..,s  ) € ^S-1 
and  for  (i,  S(y  . . . , Bj)  € J!®, 


We  can  now  state  the  main  theorem. 

Theorem  3.8.  Assume  the  following  conditions  hold: 

1.  C(i)  is  nondecreasing  in  i for  0 < i < I. 

2.  B(i)  is  nonnegative  and  nondecreasing  in  i for  0 < i < I. 

3.  A(i)  -(C(i)  + + j-  (P  . min{A(0),  C(0)  + B(0)}))  is  non- 

Mi  qi 

decreasing  in  i for  0 < i < I. 
k.  P > C(0)  + B(0). 

5.  0 < qk+1  < qk  < 1 for  0 < k < 1-1. 

6.  pi(- ) c pi+1(‘ ) por  0 < i < 1-1. 

Then  for  each  0 < a < 1 there  exists  a stationary  i control  limit 
policy  minimizing  the  total  expected  a-discounted  cost  for  the  model 
with  spare  units. 


Proof;  Note  that  Lemmas  3.5  through  3.7  hold  since  3 of  this  theorem 


implies  5 of  Lemma  3*5  with  the  assistance  of  conditions  1,  2,  4,  and 
5,  and  all  the  conditions  of  these  lemmas  are  satisfied. 

As  has  been  seen  twice  in  the  previous  chapter,  if  an  optimality 
of  an  i control  limit  policy  is  guaranteed  for  the  finite  stage  problem, 
the  result  can  be  easily  extended  for  the  infinite  horizon  case.  More- 
over, the  stationarity  of  the  i control  limit  policy  optimizing  the 
total  a-discounted  cost  is  guaranteed.  For  the  n-stage  problem,  it  is 

g 

enough  to  show  that  for  (Sq,...,s^)  6 , 0 < i < 1-1  and  n > 1, 


f jj( Sq,  . . . , Sj ) < f i+1,  Sq,  . . . , s j ) , 


where 


I I 

fn^i,sO,,,,,sI^  = + £ B(j)s^  + a E pii.Ra(i',s0,...,sI;n-l) 

I 

- (C(i)  + I B(j)s  + B(i) 

3=0  J 


+ 0:R^(0,  s q,  ... ) 


s^l,  ...,Sl;n-l)). 


Now,  for  0 < i < 1-1,  ( sn, . . . , sT)  £ ^ and  n > 0 


A ( i+1)  - C(i+1)  - B(i+1)  - (A (i)  - C(i)  - B(i) ) 

I I 

+ ^i’^O  Pi+l.i'Ra^1',S0,‘”,Sl;n)  " i,^Q  Pii'Ra(i’'S0- 


-o(Ea(0,s0,...,  s^^_^+l, . . . , s n)  - Rq/O,  Sq,  . . . ) s^+1, . . . , n) ) 


55 


■ v>  ■ 


1WU  jgiu  iu« XflllLiiiHUIJJIUII.liP.MI1  JU  l|ig,"JJUH!i IUUL IM 


Here, 


lt^ 


’ ; S j $ ft)  ^ ^ ^ } 


because  of  Lemma  3-5>  Lemma  1.5  and  condition  6.  By  using  (3.4),  "the 
last  term  can  be  simplified  to 


a(Ra(0,  Sq f • . o f s •••>Sjjn)  - R^(0,  s^,  . , s^+1, . . . , s^;n) ) 


< aU-qJM1  + a(q.  -q. , , )u1+1 
- ' a i i+1'  a 


< (l-qi)M^  + (qi-qi+1)ui+1  by  5 and  since  < M*,  U^+1< 


i „ wi  TTi+l  TTi+l 


!-q. 


- (B(i+1)  - B(i)  + (qi-qi+1)uj+1)  + (q.-q.+1)u[ 


4+1 


1_qi  Vqi+1 

(B(i+1)  - B(i) ) + 1 


qiqi+l 


(P  + B(i+1)  - min{A(0),C(0)  +B(0)}) 


Vl 


- 1 ) B(i+1) 


- ct  - 0 


B(i) 


1 — ) (P  - min{A(0),  C(0)  + B(0)})  . 


qi+l  qi 


Therefore, 

fn+i(1  + 1>so>  ■ • ^sj)  “ fn+i^i’so,,*,,sl^ 


> A(i+1)  -C(i+1)  - B(  i+1)  - (A(i)  - C (i)  -B(i)) 

B(i) 


- Ifc  - 0 B(in’  - Ct  - 0 


(~^  - (P  - rnin(A(0),  C(0)  + B(0)})J 
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:-1 


= A(i+1)  - (C(i+1)  B(i+1)  +-i — (p  - mln(A(0),C(0)+B(0)})) 


*i+l 


i+1 


- j^A(i)  - (C(i)+  j-  B(i)  +~  (P-min{A(0) , C(0)+B(0) } ) ' "| 


> 0,  by  3. 


, S 

This  last  inequality  implies  that  for  each  n > 1 and  s = (sn,...,sT)  € £ , 


there  exists  an 

i such 

s,  n 

that 

. . . , s_)  < 0 

for  any 

( If  SQ>  * * # 9 

with 

i < 

t/n^'V 

. . . , s-j.)  > 0 

for  any 

( i y Sq  y • • • J S ^ ) 

with 

i > 

Therefore,  at  the  beginning  of  the  n-stage  a-discounted  problem  (n  > 1), 
the  decision  to  repair  an  operating  machine  if  and  only  if  its  condition  i 


is  greater  than  or  equal  to  some  critical  value  i , where  (i,s)  = 

s,n’  ’ ' 


(i> Sq, . . . , s^)  is  the  state  of  the  system,  is  optimal.  This  is  an  i 
control  limit  policy  for  the  finite  stage  problem.  A similar  argument 
to  that  presented  in  Theorem  2.2  gives  us  the  rest  of  what  we  need.  □ 


In  the  previous  chapter,  we  treated  the  model  where  there  are 
no  spare  units  and  no  explicit  penalty  cost.  In  order  to  show  the 
existence  of  a control  limit  policy  minimizing  the  total  a-discounted 
cost,  a condition  was  needed  stating  that  the  operating  cost  must  increase 
more  than  the  increase  of  the  combined  material  and  expected  labor  cost 
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■ 

i 

{ 


i 


for  repairing  a machine.  Here,  because  of  the  presence  of  spare  units 
and  the  explicit  penalty  cost,  a stronger  condition  is  required  as  is 
seen  in  condition  3.  Since  P - min{A(0),  C(0)  + B(0)}  can  be  considered 
as  a relative  value  of  the  penalty  cost  per  period,  3 can  be  interpreted 
as  follows:  the  operating  cost  must  increase  more  than  the  increase  of 

the  combined  material  and  expected  labor  costs  and  the  maximal  expected 
penalty  cost  effect  for  a single  repair  of  a machine.  Notice  that  the 
penalty  effect  vanishes  if  the  repair  time  distribution  does  not  depend 
on  the  condition  of  the  machine  to  be  repaired,  i.e.,  if  qfc  = q for 
any  k.  Other  conditions  are  not  very  restrictive.  From  1 and  2,  it 
is  evident  that  the  worse  the  condition  that  an  operating  machine  is  in, 
the  more  expensive  both  the  material  and  labor  costs  are  for  repairing 
it.  From  5,  the  worse  the  condition  that  an  operating  machine  is  in, 
the  longer  the  expected  repair  time  is.  4 requires  that  the  penalty 
cost  is  at  least  as  expensive  as  the  cheapest  total  repair  cost  of  a 
single  machine.  The  IFR  property  is  guaranteed  by  6. 

The  long-run  average  cost  version  of  Theorem  3.8  can  be  easily 
obtained  as  before: 

Theorem  3.'.-*.  Assume  that  all  the  conditions  in  Theorem  3.8  hold,  and 
furthermore,  suppose  that  any  operating  machine  eventually  fails.  Then 
there  exists  a stationary  i control  limit  policy  minimizing  the  long-run 
expected  average  cost  of  the  model  with  spare  units. 
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Proof , If  any  operating  machine  eventually  fails,  then  Theorem  1.4 


holds  since  (i,  sQ, . . . , s^.)  = (0,0,,..,  0)  is  accessible  from  every  other 
state  no  matter  which  stationary  policy  is  employed.  The  rest  of  the 
proof  is  similar  to  that  of  Theorem  2.5,  and  hence,  can  be  omitted.  □ 


5.2.  Penalty  Cost  and  Leasing  Options 

A penalty  cost  has  been  introduced  in  this  chapter  to  distinguish 
the  system' s failure  from  the  state  where  the  system  is  operating. 

Notice  that  a system  may  also  fail  for  the  models  without  spare  units. 

The  reason  why  this  penalty  cost  was  not  considered  in  Chapter  2 is  that 
in  those  models  the  system’s  failure  coincides  with  the  machine's  failure, 
and  hence,  the  penalty  cost  for  the  system' s failure  can  be  included  in 
the  labor  cost  for  repairing  the  machine.  For  the  model  with  spare  units, 
the  failure  of  an  operating  machine  does  not  necessarily  mean  the  system's 
failure,  and  so  this  undesirable  condition  must  be  treated  separately 
from  other  states,  which  requires  the  introduction  of  such  a penalty 
cost. 

If  it  is  very  undesirable  to  discontinue  an  operation,  or  a 
penalty  cost  for  the  system's  failure  is  extremely  expensive,  and  such 
a situation  should  be  avoided,  then  a possible  alternative  would  be  to 
obtain  a machine  on  lease  from  a leasing  company  when  the  necessity 
comes  up.  Suppose  getting  a machine  on  lease  is  allowed  only  when  the 
system  fails,  and  the  cost  for  the  lease  is  D per  period.  A leased 
machine  continues  to  operate  at  its  best  condition  during  a leasing 
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period  since  the  leasing  company  has  the  responsibility  to  replace 
older  machines  with  new  ones  at  its  own  expense  whenever  an  operating 
leased  machines  become  impercet.  Therefore,  the  cost  D includes 
the  cost  to  maintain  an  operating  machine  in  its  best  condition,  and 
hence,  it  can  be  expensive..  Thus,  a decision  to  obtain  a machine  on 
lease  would  be  practical  only  when  the  system  fails. 

Now  consider  the  previous  model  with  an  option  of  leasing  as 
stated  above.  The  dynamic  programming  formulation  is  almost  identical 
with  an  exception  that  every  P which  appeared  in  the  formulation 
is  replaced  by  min{P,D).  Hence,  it  is  clear  that  if  P > D then 
obtaining  a machine  on  lease  is  optimal  whenever  the  system  fails 
and  if  P < D then  leasing  had  better  be  forgotten. 

In  the  above  discussion  the  leasing  option  is  accepted  only 
when  the  system  fails.  Now  we  can  show  that  this  statement  is  not 
an  assumption  to  be  made,  but  is  a rule  which  an  optimal  policy 
must  follow.  To  see  this,  let  us  allow  the  possibility  that  the 
leasing  option  is  eligible  at  any  time  period.  If  the  leasing  option 
is  chosen  when  there  is  an  operating  machine,  it  is  practical  to 
think  that  the  operating  machine  replaced  by  a leased  machine  is 
brought  to  a repair  shop  no  matter  how  good  it  is.  This  gives  a dynamic 
programming  formulation  as  below:  For  n > 1, 
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Obviously  for  i c J) , (Sg,...,^)  £ ^S,  the  third  term  which 
gives  the  minimum  n period  cost  when  the  leasing  option  is  chosen  at 
the  beginning  is  no  less  than  the  second  term  if  D > 0,  and  hence,  an 
optimal  policy  has  the  property  that  a leasing  decision  is  not  selected 
whenever  there  is  an  operating  machine. 

6l 


CHAPTE.F  U 


COMPUTATIONAL  RESULTS 

This  chapter  treats  the  problem  of  finding  the  optimal  cost 
and  its  corresponding  policy  knowing  that  an  optimal  policy  is  of  a 
control  limit  form.  Some  recursive  algorithms  are  presented  for  the 
total  a-discounted  cost  case.  For  the  long-run  average  cost  case,  an 
explicit  calculation  of  an  optimal  control  limit  policy  is  obtained  for 
systems  with  simple  structures.  In  the  last  section  some  concluding 
remarks  are  given,, 

4,1.  Algorithms  for  Calculati ng  the  Optimal  Cost 

As  has  been  seen  before,  each  model  treated  in  this  paper  can 
be  formulated  as  a Markov  decision  process.  Therefore,  the  usual 
recursive  algorithms  to  calculate  the  optimal  policy  for  the  total 
a-discounted  cost  or  the  long-run  average  cost  can  be  applied  to  our 
models.  These  algorithms  are  mainly  policy  improvement  procedures  and 
linear  programming  approaches.  Since  these  general  techniques  are 
studied  in  many  papers,  for  example  in  [4],  [8],  [l6],  [19],  we  will 
not  discuss  them  in  detail  here.  In  our  models  an  optimal  policy  is 
assured  to  be  of  a control  limit  form  if  certain  conditions  are 


satisfied.  Knowing  that  an  optimal  policy  is  of  a control  limit  form, 

we  can  expect  that  better  algorithms  exist  since  tni;  information 

should  enaeie  as  to  explore  this  struct  ire,  thereby  decreasing 

significantly  the  numoer  of  policies  tnat  must  be  considered. 

The  model  treated  here  is  the  general  Markov  decision  model 

defined  in  Sec  on  1,3  with  the  action  space  A = {0.1}  0 represents 

the  action  to  keep  a machine  in  operation,  and  1 represents  the  action 

to  repair  it.  Consider  the  total  '-discounted  cost  problem.  Suppose 

an  optimal  policy  is  known  to  be  of  a control  limit  form.  Then  by 

letting  R.  be  a control  limit  policy  with  a control  limit  j,  the 
d 

R.'s  (0  < j < I)  are  the  candidates  for  the  optimal  policy.  A simple 
*D 

enumeration  is  one  possible  method  to  find  an  optimal  control  limit 

policy  though  it  does  not  seem  to  be  very  attractive.  Total  scanning 

is  not  necessary  if  the  following  lemma  is  used  since  it  tells  us 

that  once  V is  obtained,  the  neighboring  two  values  V and 

V hk+l,a 

VD  can  be  easily  checked  to  determine  whether  or  not  they  are 

V V 

better  than  V„  Without  having  any  ambiguity,  we  write  VD  for 

Va  " \ 

V_  for  the  rest  of  this  section  and  the  next  section 

Fk,a 

Lemma  1.1.  The  following  three  expressions  are  equivalent 
For  i - 0,  1,...,  1-1, 


<a)  VR.?VR.  , 
l i '-1 


I 


(b)  C(i.O)  - a L p.  (0)V  (j)  - C ( i , 1 ) + a L p..(l)V  (d). 

J=0  Ri  j"0  R1 


I 


I 


vc)  Cfi.O)  +a  Z p.  /0)V  (j)  r C(i.l)  + a Z p,,(l)V  (,j)  .. 

j-0  1J  Ritl  " ij  Ri+i 


Proof;  We  show  that  (a)  and  (b)  are  equivalent.  The  equivalence 
between  (a)  and  (c)  can  be  similarly  shown. 

A control  limit  policy  fh  ^ (f^f  , ...)  has  the  form 


0 for  0 < j < i 

1 for  i < j < I, 


and  from  Markov  decision  theory,  VR  satisfies  the  following  eauation. 

i 

I 

vR  (J)  = c(j,f  (j))  + a Z p (f.(j))v  (j’),  o < j < i.  (4.1) 

i j'=0  i 

Now  let  CR  (j)  = C(j,f.(j)),  0 < j < I,  and  let  PR  be  the  transition 
i i 

probability  matrix  modified  by  the  policy  R . Then  we  can  solve  (4.1) 


for  VR  and  obtain 
i 


VR.  = -QPR.>  CR.  • 
1 11 


Using  this  expression  and  from  simple  calculations,  we  have 

I 

VP"^R  «=  fC(i,l)  - C(i,0)  + a(  E (p.,(l)  - P (0))V  (j))] 
i i+1  j=o  ^ ^ Ri 

* [(i+l)-st,  column  of  (i  - aP  )-1]  . 


As  each  column  of  (i  - aF  ) is  nonnegative  and  has  at  least 

i+1 

positive  element,  the  claim  follows.  □ 


Definition.  VD  is  said  to  be  unimodal  in  i (0  < i < I')  if 
i “ 

VR  < vn  implies  V < V (2  < i < I)  and  Vn  < V„ 
i Ri-1  Ri-1  Ri-2  " " Ri  Ri+l 

implies  V < V (0  < i < 1-2). 
l+l  Ri+2 


is  unimodal  in  i,  then  Fibonacci  type  search  can  be 


applied  resulting  in  a faster  convergence  rate 


Folicy  improvement  procedures  car,  also  be  improved  if  V. 


is  unimodal  in  i.  If  V 


is  unimodal  in  i,  then  the  technique 


of  the  policy  improvement  algorithm  can  be  performed  among  the  set 


of  control  limit  policies  rather  than  the  set  of  whole  stationary 


policies..  Therefore  the  convergence  rate  of  the  special  policy  improve 


ment  technique  shown  next  should  be  much  faster  than  the  general  policy 


.improvement  technique.  The  algorithm  is  as  follows 


Step  0.  Find  a control  limit  policy  R 


Step  1.  Compute  the  value  V by  solving  (4.1)  or  for  0 < i < I 


(/,  then  R.  is  optimal.  Otherwise,  a control 


limit  policy  R 


is  strictly  better  than  R, 


Go  back  to  Step  1 where  R 


Notice  that  if  R 


is  not  optimal,  then  either  V. 


By  Lemma  k. 1,  the  former  leads  to  i -1  £ S.,  and 


Hence,  if  R.  is  not  optimal,  then 


Also  notice  that  if  i 


S U S , then  by  the  definition 


Using  the  monotonicity  of  T. 


we  have 


Therefore,  R.  is  a strict  improvement 


The  problem  thus  becomes  that  of  determining  under  what  con 


ditions  the  unimodality  of  V is  assured.  The  following  theorem  holds 


Theorem  4,2.  VR  is  unimodai  in  i (0  < i < I)  under  the  following 


conditions 


Proof:  First  notice  that  the  first  three  conditions  guarantee  the 

existence  of  a control  limit  policy  optimizing  the  model. 


We  want  to  show  that  V-,  > implies  > V_  for 

Ri+1  Ri  Ri+2  Ri+1 

0 < i < 1-2.  Suppose  VD  > V_,  . Then  for  ,j  > i+1, 

n.  , ri.  — 

1+1  i 


V (j)  = c ( j , l ) + « Z p (1)V-  (k),  by  (4.1) 

i+1  krO  i+l 

I 

£ C(j+l,l)  + a Z p (l)V  (k)  , from  4 and  1 
k-0  H±+l 

= VR  (j+1),  by  (4.1) . 
i+1 


Hence,  VR  (j)  is  nondecreasing  in  j for  j > i+1.  Furthermore, 
i+1 

V (i)  - V (i+1) 

Ri+1  Ri+1 

I I 

= c(i,0)  + a Z p . . (o) v (j)  - (C(i+1,D +a  S p. . (i)v  (j)) 
j=0  1J  Vl  jr-0  1+1>J  Ri  + 1 


< C(i,0)  + a Z p..(0)V  (j)  - (C(i,l) +a  Z p.,(l)V_  (j)), 

j-0  Vl  j=o  13  Ri+1 


by  1 and  4 


= 9 > 0,  from  VR  > VR  and  by  Lemma  4.1(a), 
l+l  l 


C(i+1,0)  +■  a Z p.  (0)V  (j)  - (c(i,0)  + a Z p..(0)VR  (j)) 

d--0  1+1,3  Ki+1  j*o  13  Ri+1 

I 

“ C(i+1,0)  -C(i,0)  +a  Z p (0)V  (d)-a(p  (0)V  (i+1) 

d-i+1  1 '’J  Ri+1  11  Ri+1 

+ Pi  i+l(0)VR  (i+1)  i+P(0)VR  (i+2)  + •*•  + PiT(0)VR  (I)) 
1,1  1 Ri+1  1,1+2  Ri+1  11  Ri+i 


a(Pii(°)vR_  (i)  - Pii(0)VR  (i+l)) 


1 


> C(i+1,0)  - C(i,0)  - ap  .(0)(V_  (i)  - v (i+1)), 

11  Ri+1  Ri+1 

since  VR  (i+1),  V (i+1),  V (i+2),...,V  (i)  i 


Ri+1 


Ri+1 


R.  , 

i-t-1 


is 


i+1 


nondecreasing,  and  by  5 
> C(i+1,0)  - C(i,0)  - OB. 

Finally, 

I I 

C(i+i,0)+ a Z p...  . (o)v  (j)  - (c (i+l, l)  + a Z p.+1  .(l)v  (j)) 

j=0  1+1^  Ri+1  0=0  1+1’J  Ri+1 

I I 

= (C(i+1,0)  + a Z p.  , ,(o)v  (d))  - (c(i,0)  +a  Z p,,(o)v  (j) ) 

o-o  1 1,J  Ri+i  o=o  1J  Ri+i 

i i 

+ (c(i,o)+a  Z p.  . (o )v  (d ) ) - (c(i, l)  + a Z p.,(i)V  (j)) 

0--O  Ri+1  0=0  1J  Ri+1 

+ C(i,l)  - C(i+1, 1) , by  k 

> C(i+1,0)  - C(i,0)  - 08  + B + C(i, 1)  - C(i+l,l)  > 0,  from  2. 


Therefore,  by  Lemma  4.1(a'),  we  have  V > V 

Ri+1  Ri+1 


In  a similar 


manner  we  can  prove  that  V < V_,  implies  V < V_  , which 

Ri  Ri-1  Ri-1  Ri-2 

completes  the  proof  of  unimodality  of  V in  i (0  < i < I).  □ 

K . — — 

1 


Condition  5 says  that  a machine  only  deteriorates  and  never 

gets  better  while  it  keeps  operating,  which  seems  to  be  a reasonable 

assumption.  Condition  h is  satisfied  for  the  classical  replacement 

models  since  p^Q(l)  = 1 for  any  i.  Also,  in  our  model  treated 

in  Section  2.1,  condition  4 is  automatically  satisfied.  Further,  if, 

in  addition  to  the  conditions  in  Theorem  2.2,  we  assume  p. , = 0 for 

ij 

any  j < i-1,  the  unimodality  of  V in  i is  guaranteed.  For  other 

Ri 

models,  condition  4 seems  too  restrictive  since  it  requires  the  repair 
time  distribution  of  a machine  to  be  independent  of  its  operating 
condition.  gg 


4.2.  Policy  Improvement  Hybrid  Method 


I I 


< 

I 


When  a policy  improvement  technique  is  employed,  the  number  of 
feasible  policies  can  be  significantly  reduced  if  better  policies  are 
searched  recursively  among  a set  of  stationary  control  limit  policies. 
As  has  been  just  seen,  this  can  be  achieved  if  unimodality  of  V in 

n . 

1 

i is  assured,  and  unimodality  is  satisfied  under  mild  conditions  for 
replacement  or  simple  structured  repair  models.  For  more  complicated 
models,  unimodality  can  no  longer  be  expected.  Then  an  optimal  policy 
may  or  may  not  be  obtained  iteratively  by  searching  better  policies 
among  a set  of  stationary  control  limit  policies.  But  it  is  con- 
ceivable to  think  that  we  can  still  proceed  efficiently  if  we  construct 
an  algorithm  where  "good"  policies  are  searched  iteratively  among 
stationary  control  limit  policies  whenever  possible  (first  phase), 
and  then  we  switch  to  a usual  policy  improvement  among  stationary 
policies  (second  phase).  The  second  phase  starts  with  a policy  which 
is  better  than  the  control  limit  policy  obtained  at  the  end  of  the 
first  phase  if  the  first  phase  does  not  yield  an  optimal  policy.  Thus 
a hybrid  algorithm  is  introduced  and  is  indicated  in  the  flow  chart 
of  Fig.  4. 1. 

Step  0.  Find  a control  limit  policy  R?  = (f.  ,f.  ,...). 

10  1o  1o 


Step  1.  Compute  the  value  V^p  = V by  solving 


Vf  (j)  = C(j,f  (j))+a  Z (j))V  (j'),  0 < j < I. 

i0  0 j'=0  x0  xlQ 


Step  2.  Let 


\ = CO  < i'  < i0|c(i,l)  + a E p (J)  < Vf_  (i) 


for  any  i with  i'  < i < i -1) 


S2  = {io  < r - Ilc(i’0)  + a E p^(°)v^.  (J)<V,  (i) 


riV  f 

0=0  1J  ir 


for  any  i with  iQ  < i < i'-l}  . 


Take  i^  to  be  the  smallest  i1  £ or  the  largest  i'  £ S 

If  S1  US  f i exists  and  r}  = (f.  ,f.  ,...)  is 

x ^ 1 X1  11  11 
0 A 1 
strictly  better  than  R.  , and  go  back  to  Step  1 where  R. 

o 1q  1 

is  now  a new  R.  . If  S,  = S = go  to  Step  3. 

0 L d 


Step  3.  Let  R = (f, f . . . ) be  such  that 


C (i, f (i) ) + a E p (f(i))V  (o) 

0=0  1J  *0 
I 

= min  {C(i,a)  + a E p .(a)V  (j) ), 

a=0,l  0=0  Ji0 


0 < i < I. 


Compute  the  value  V ? = V by  solving 

R 1 


V.(i)  = C(i,f (i) ) + a E p ,(f(i) )V „(o),  0 < i < I 


If  Vf  is  not  an  improvement,  R^  is  optimal.  If  Vf  is  an 
improvement,  then  go  to  Step  4. 


1 


? 2 2 

Step  4,  If  P • R , or  if  R is  a control  limit  policy,  then  go 
12 

to  Step  2 where  ft  is  now  a new  E.  , otnerwise  go  to 

2 X°  2 
Step  3 where  the  present  V^,  is  now  a new  V^,  and  R 

is  renewed. 

The  justification  of  the  above  algorithm  is  similar  to  that 
of  the  previous  algorithm  in  the  last  section.  The  algorithm  is 
presented  for  the  models  without  spare  units,  but  the  hybrid  algorithm 
can  be  applied  to  the  models  with  spare  units  with  a slight  notational 
change 


■ ♦ . 3 . Explicit  Calculation  of  the  Optimal  Control  Limits 

Knowing  that  an  optimal  policy  is  of  a control  limit  form, 
we  found  some  recursive  algorithms  which  produce  an  optimal  control 
limit  for  the  total  a- discounted  cost  problem.  When  we  move  on  to 
the  long-run  average  cost  problem,  we  find  that  both  policy  improvement 
techniques  and  LP  approaches  also  are  applicable  to  that  case.  Moreover, 
if  the  model  has  a simple  structure  described  as  below,  we  can  even 
calculate  the  optimal  control  limit  explicitly.,  Explicit  calculation 
makes  it  easier  to  grasp  the  effect  of  several  cost  coefficients  on 
the  determination  of  a control  limit.  In  this  section,  we  consider 
a few  examples  and  find  an  explicit  way  of  calculating  an  optimal 
control  limit  in  each  model.  We  require  simplifying  assumptions  on 
the  transition  probabilities,  the  linearity  of  several  costs,  and  the 
linearity  of  the  expected  repair  time  length.. 
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Example  4,1.  We  consider  a maintenance  model  without  spare  units. 


The  transition  probability  p. the  parameter  q.  of  repair  time 

■*o  1 

distribution,  and  the  cost  functions  A(i),  C(i)  and  B(i)  are 
defined  as  follows: 

Il-p  if  j = i 

p if  j = i+1,  0 < i < 1-1. 

0 otherwise 


q > o , o < i < i. 

A(i)  = aQ  + ai  , 0 < i < I. 

C(i)  =cQ+ci,  0 < i < I 

B(i''  = bQ  + bi  , 0 < i < I. 


! 

I 


i 

; 

\ 

I 


The  long-run  average  cost  using  a control  limit  policy  R (i 
is  its  control  limit)  can  be  calculated  as  the  ratio  of  the 
expected  cost  between  returns  to  state  0 when  R^  5s  employed  to 
the  expected  length  of  a return  to  state  0 given  is  employed. 

In  this  example,  by  letting  l/p  s A , 1/q  = ^ for  simplicity, 

. W f f i(i~1))  + co  + cl  * ^o(bo  + bi> 

V V + Mo 

Taking  the  derivative  of  V with  respect  to  i,  setting  the 

i 

value  of  the  derivative  to  be  zero,  and  solving  it  for  i (>  0)  yields. 


be  the  optimal  control  limit 


that  if  the  term  inside  of  the  square  root  of  the  above 


negative  or  if  i 


< 0,  tnen  i*  = 0„  A simple  calculation  shows  that 


< I,  then  i* 


Example  l„2  We  generalize  Example  4.1  by  letting 


Then 


The  same  analysis  as  in  Example  4.1  gives 


where 


Example  4.3 


If  we  further  generalize  Example  h.2  by  letting 


where 


\ \ 


M 

i 

i 

i i 


T-  * \ “ xo  + X1  ’ 0 S 1 £ I-1> 


then  the  expression  gets  much  more  complicated,  and  i = i is 

Opu 

the  solution  of  the  following  polynomial  of  degree  4: 


X2a  .4  , 2\jra  3 (or  Xp  , v2 

1 f — 1 + + AnAa)i 


u0c 


+ lQWr  Ac0"  AM0b0)i+  crX  ' M«rbn)  = 0 > 


0 O'  o' 


where 


a = XaQ  + XQa  - Xa  + 24b 


P = A0a0  ‘ 2 Aa0  " 2 V + Z M + C + ^b0  + »0X 


Y = A0  " 2 A + u' 


In  this  example,  the  optimality  of  a control  limit  policy  is 
guaranteed  if  c,  b and  4 are  nonnegative,  cQ  •-  0 and 


> c + 4bQ  f 4()b  + (21  - l)ub. 


Example  4.4.  We  generalize  Example  4,1  by  allowing  an  operating  machine 

to  transit  into  a failed  state  in  a period.  In  this  example,  an 

operating  machine  will  fail  to  operate  in  one  period  with  probability  r 

(>  0),  and  if  not,  then  it  goes  to  the  next  worse  state  with  probability  p, 

or  it  remains  in  the  same  state  with  probability  1-p. 

An  optimality  of  a control  limit  policy  is  guaranteed  if 

a > c > 0.  Assuming  this  condition,  it  is  not  hard  to  see  that  V„ 

Ri 

is  given  by  the  following  expression; 
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if  I 

. ' -r  ■ I 

7i  1 

a.-v  ; >•  I 


7 t aQ(l-d) 1 + a(^§-  - (i-l)d1)]  + (c(i-l)  + £ (i-I) ) 


^ t . TUa^ 


where 


- (l-d1)  + - 

r q 


0 < d = — * * < 1 . 

- p + r - pr 


Taking  the  derivative  of  V with  respect  to  i and  setting 

K. 

1 


it  to  be  zero  yields. 


s - (-  - r)di 


(log  d)(j  + -)(--  r)i  - (7  - r)(|  + |) 


(log  d)  (— + 

0 ' v qr  r 


+ El)  = 0, 

q • > 


where 


t b .0 

r 1=0  + \>  ro  = co  T 


If  a/r  > y,  we  can  show  that  dfi(i)/di  > 0.  Then  £2(i)  has 
at  most  one  zero  point,  and  if  we  let  ^(^0pt)  = 0 when  there  is  a 
zero  point  of  fi(i),  i*  = [ io^^.  ] or  [ i ^ + 1]  for  0 < i ^ < I, 
i*  = 0 for  i < 0,  and  i*  = I for  i , > I,  If  there  is  no 
zero  point,  then  i*  = 0. 

If  the  failure  rate  r is  small  enough,  then  d is  close  to  1, 
and  hence,  we  can  approximate  the  zero  point  of  12 (i)  by  using  the 
first  degree  of  approximation  dd  = e^log  1 = 1 + (log  d)i.  Then, 


opt  log  d 


raQ  - rqy0  + p(l-r)a  + r rl 
r(a  - it) 


4.4.  Conclusion 


As  was  noted  in  the  previous  chapters,  optimality  of  a control 
limit  policy  is  guaranteed  under  certain  mild  conditions  in  each  main- 
tenance model  with  repair.  The  main  restrictions  are  placed  on  the  cost 
functions.  In  the  case  of  replacement  models,  the  nondecreasing  property 
of  the  difference  of  the  operating  cost  and  the  material  cost  is  required. 
If  the  repair  models  without  spare  units  are  of  interest,  the  nondecreasing 
property  of  the  difference  of  the  operating  cost  and  the  total  repair 
cost  for  a single  repair  work,  including  the  material  and  labor  costs 
is  required.  When  the  repair  model  with  spare  units  is  treated,  this 
condition  is  strengthened  in  the  sense  that  the  total  repair  cost  now 
includes  the  effect  of  the  penalty  cost  as  well  as  the  material  and 
labor  costs. 

For  the  model  with  spare  units,  we  assume  that  there  are  a 
large  enough  number  of  repairmen  in  the  repair  shop  so  that  the  repair 
work  can  begin  immediately  on  any  machine  sent  to  the  repair  shop. 

Having  only  one  repairman  in  the  repair  shop  also  seems  to  be  a reason- 
able assumption.  Then  at  most  one  machine  can  be  repaired  at  a time, 
and  if  more  than  one  machine  is  to  be  repaired,  they  must  form  a 
queue  while  they  wait  for  service.  This  modified  problem  results  in 
a slightly  different  formulation,  but  it  can  be  shown  that  almost 
identical  conditions  as  presented  in  this  paper  will  be  sufficient 
for  the  optimality  of  an  i control  limit  policy. 


77 


1. 

2. 

5. 

b. 

5. 

6. 

7. 


8. 


9. 


10. 

11. 


12. 


13. 


I1*. 


,11,  ..HI' W};, 


REFERENCES 

Barlow,  R.E.  and  F.  Proschan,  Statistical  Theory  of  Reliability  and 
Life  Testing,  Holt,  Rinehart  and  Winston,  1975. 

Bellman,  R.,  Dynamic  Programming,  Princeton  University  Press, 
Princeton,  New  Jersey,  1957 » 

Blackwell,  D.,  "Discrete  Dynamic  Programming,"  Annals  of  Mathematical 
Statistics,  33  (.1962),  pp.  719-726. 

Dantzig,  G.B,  and  P.  Wolfe,  "Linear  Programming  in  a Markov  Chain," 
Operations  Research, 10  ( 1962 ) , pp . 702 -7 10 . 

Derman,  C.,  "On  Sequential  Decisions  and  Markov  Chains,"  Management 
Science,  j?  (1962),  pp.  l6-2b. 

Derman,  C.,  "Optimal  Replacement  and  Maintenance  Under  Markovian 
Deterioration  with  Probability  Bounds  on  Failure,"  Management  Science, 
2 (1963),  pp.  478-481. 

Derman,  C.,  "On  Optimal  Replacement  Rules  when  Changes  of  State  are 
Markovian,"  in  Mathematical  Optimization  Techniques,  R.  Bellman  (ed). 
University  of  California  Press,  Berkeley,  1963. 

Derman,  C.,  Finite  State  Markovian  Decision  Processes.  Academic 
Press,  New  York,  1970. 

Derman,  C and  G.J.  Lieberman,  "A  Markovian  Decision  Model  for  a 
Joint  Replacement  and  Stocking  Problem,”  Management  Science,  13,  No,  9 

(1967),  pp.  609-617. 

Eppen,  G D,,  "A  Dynamic  Analysis  of  a Class  of  Deteriorating  Systems," 
Management  Science,  12,  No,  3 ( 1965 ) » PP-  223-2* 0. 

Kalymon,  B.A.,  "Machine  Replacement  with  Stochastic  Costs," 

Management  Science,  18,  No.  5 (1972),  pp.  288-298. 

Kao,  E.,  "Optimal  Replacement  Rules  when  Changes  of  State  are 

Semi -Markovian, " Operations  Research,  21,  No.  6 (1973)>  PP-  1231-12*. 9. 

Klein,  M. , "Inspection-Maintenance-Replacement  Schedules  Under 
Markovian  Deterioration,"  Management  Science,  9 (1962),  pp.  25-32. 

Kolesar,  P.,  "Minimum  Cost  Replacement  Under  Markovian  Deterioration," 
Management  Science,  12,  No,  9 (1966),  pp.  694-706. 


Kolesar,  P.,  "Randomized  Replacement  Rules  which  Maximize  the 
Expected  Cycle  Length  of  Equipment  Subject  to  Markovian  Deterioration," 
Management  Science,  13,  No,  11  (196?),  pp,  867-876. 

Manne,  A.,  "Linear  Programming  and  Sequential  Decisions,"  Management 
Science,  6 (i960),  pp,  259“267. 

Rosenfield,  D,B„,  "Deteriorating  Markov  Processes  Under  Uncertainty," 
Technical  Report  No,  162,  Department  of  Operations  Research  and 
Department  of  Statistics,  Stanford  University,  Stanford,  1974. 

Ross,  S„,  "A  Markovian  Replacement  Model  with  a Generalization  to 
Include  Stocking,"  Management  Science,  15,  No,  11  (1969),  pp.  702-715. 

Ross,  S. , Applied  Probability  Models  with  Optimization  Applications, 
Holden-Day,  San  Francisco,  1970. 

Taylor,  H.M. , "Markovian  Sequential  Replacement  Processes,"  Annals 
of  Mathematical  Statistics,  36  (1965),  pp,  1677  "3-694. 

Wagner,  H.M, , "On  the  Optimality  of  Pure  Strategies,"  Management 
Science,  6 (i960),  pp,  268-269 

Wagner,  J„V,,Jr.,  "Maintenance  Models  for  Stochastically  Failing 
Equipment,"  Technical  Report  No.  153,  Department  of  Operations 
Research  and  Department  of  Statistics,  Stanford  University, 

Stanford,  1973, 


UNCLASSIFIED 


SdCUNITY  CLASSIFICATION  OF  THIS  PACe  (TFAfn  DaIa  Bai ...d; 


20.  Abstract 


In  this  study  discrete  time  finite  state  Markov  maintenance  models  are 
investigated.  In  each  model,  a machine  is  assumed  to  be  operating  over  tim 
with  its  condition  deteriorating  as  time  goes  on.  The  state  of  the  machine 
is  observed  at  the  beginning  of  a period.  An  operating  machine  can  be 
sent  to  a repair  shop  at  this  time,  whereas  a failed  machine  must  be 
repaired.  When  a machine  is  being  repaired,  the  number  of  time  periods 
that  it  is  unavailable  is  usually  assumed  to  have  a geometric  distribution, 
A repaired  machine  becomes  available  in  its  best  state.  An  operating  cost 
is  charged  while  a machine  is  operating,  and  material  and  labor  costs 
are  charged  when  it.  is  being  repaired.  The  objective  is  to  find  a policy 
which  minimizes  the  total  expected  a-discounted  cost  or  the  long-rung 
average  cost.  Special  emphasis  is  being  placed  on  finding  sufficient 
conditions  to  assure  that  a control  limit  policy  is  optimal.  The 
aforementioned  model  had  only  one  machine  in  the  system.  Models  with 
spare  machines  in  the  system  are  next  studied.  For  these  models  a penalty 
cost  is  added  when  the  system  fails  (only  when  all  machines  are 
inoperative) . 
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